Method, device, equipment and storage medium for generating reference image of ultrasonic image

Abstract

The present application relates to the field of image feature processing, and in particular to a method, device, computer equipment and storage medium for generating a reference image of an ultrasound image, the method comprising: obtaining pre-configured mask information; obtaining an original ultrasound image; multiplying the original ultrasound image with the mask information to obtain a feature region ultrasound image; inputting the feature region ultrasound image into a deep learning network to obtain a first feature variable; inputting the original ultrasound image into a deep learning network to obtain a second feature variable; obtaining the shape feature of the image according to the first feature variable, and obtaining the fine-grained feature of the image according to the second feature variable; generating a reference image according to the shape feature and the fine-grained feature. The present application can generate an accurate and reliably interpretable reference image, thereby improving the reliability of the prediction results.

Description

Technical Field

[0001] The present application relates to the field of image feature processing, and in particular to a method, device, computer equipment and storage medium for generating a reference image of an ultrasound image. Background Art

[0002] With the development of imaging technology, digital images have become the main data of medicine, and image recognition is performed through artificial intelligence to assist doctors in clinical decision-making. The biggest obstacle to the penetration of artificial intelligence technology into the medical field currently stems from the "black box" problem of deep neural networks. Humans cannot trust the decisions made by unexplainable AI because they cannot accurately and reasonably explain AI decisions. At present, one method is to calculate the contribution of ultrasound images to the prediction results of the network through zero-value maps. Since accurate reference images cannot be provided, this method cannot determine whether the network focuses on the shape features or texture features of the ultrasound image, and thus cannot determine whether the network focuses on the correct features. That is, the current reference map does not have good ultrasound reference map characteristics and cannot accurately represent the feature information in the image. Summary of the invention

[0003] The main purpose of the present application is to provide a method for generating a reference image of an ultrasound image and a projection connection control method, device, computer equipment and storage medium, aiming to solve the problem of low accuracy of feature representation of ultrasound reference images.

[0004] In order to achieve the above-mentioned object of the invention, the present application proposes a method for generating a reference image of an ultrasound image, comprising:

[0005] Get pre-configured mask information;

[0006] Acquire original ultrasound images;

[0007] Performing a product process on the original ultrasonic image and the mask information to obtain an ultrasonic image of the characteristic region;

[0008] Inputting the characteristic region ultrasound image into a deep learning network to obtain a first characteristic variable; inputting the original ultrasound image into a deep learning network to obtain a second characteristic variable;

[0009] Obtaining a shape feature of the image according to the first feature variable, and obtaining a fine-grained feature of the image according to the second feature variable;

[0010] A reference image is generated according to the shape feature and the fine particle feature.

[0011] Furthermore, the step of inputting the characteristic region ultrasound image into a deep learning network to obtain a first characteristic variable; and inputting the original ultrasound image into a deep learning network to obtain a second characteristic variable comprises:

[0012] Inputting the characteristic region ultrasound image into a deep learning network, transforming the latent code of the characteristic region ultrasound image based on a nonlinear mapping network and an affine transformation in the deep learning network, and obtaining a first characteristic variable;

[0013] The original ultrasound image is input into a deep learning network, and the latent coding of the original ultrasound image is transformed based on a nonlinear mapping network and an affine transformation in the deep learning network to obtain a second feature variable.

[0014] Further, obtaining the shape feature of the image according to the first feature variable and obtaining the fine-grained feature of the image according to the second feature variable includes:

[0015] Inputting the first feature variable into a low-resolution convolutional layer in a generative network to obtain shape features of the image;

[0016] The second feature variable is input into the high-resolution convolutional layer in the generative network to obtain fine-grained features of the image.

[0017] Furthermore, the low-resolution convolution layer has a resolution of 4 2 -32 2 The high-resolution convolution layer has a resolution of 64 2 -1024 2 The convolutional layer.

[0018] Furthermore, after generating the reference image according to the shape feature and the fine-grain feature, the method further includes:

[0019] Inputting the original ultrasound image and the reference image into a classification network, and calculating the increment of the original ultrasound image compared to the reference image based on the classification network;

[0020] An attribution graph is generated based on the increments.

[0021] Furthermore, the step of inputting the original ultrasound image and the reference image into a classification network, and calculating the increment of the original ultrasound image compared to the reference image based on the classification network, includes:

[0022] Dividing the original ultrasound image into a plurality of regional sub-images;

[0023] Inputting each of the region sub-images and the reference image into a classification network respectively, and calculating a first contribution of each of the region sub-images to a prediction score of a respective result compared to the reference image based on the classification network;

[0024] Acquire a second contribution of the environmental feature in the reference image;

[0025] The increment is determined according to the first contribution and the second contribution.

[0026] Furthermore, after generating the attribution graph according to the increment, the method further includes:

[0027] Obtaining the target sub-region image with the highest contribution in the attribution map;

[0028] The effective feature information of the reference image is determined according to the target sub-region image.

[0029] The present application also provides a device for generating a reference image of an ultrasound image, comprising:

[0030] A configuration information module, used to obtain pre-configured mask information;

[0031] An original image module, used for acquiring original ultrasound images;

[0032] An image processing module, used for performing product processing on the original ultrasonic image and the mask information to obtain an ultrasonic image of a characteristic region;

[0033] A variable conversion module, used for inputting the characteristic region ultrasound image into a deep learning network to obtain a first characteristic variable; and inputting the original ultrasound image into a deep learning network to obtain a second characteristic variable;

[0034] A feature extraction module, used for obtaining a shape feature of the image according to the first feature variable, and obtaining a fine-grained feature of the image according to the second feature variable;

[0035] A reference image module is used to generate a reference image according to the shape feature and the fine particle feature.

[0036] The present application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method for generating a reference image of an ultrasound image described in any one of the above items are implemented.

[0037] The present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method for generating a reference image of an ultrasound image described in any one of the above items are implemented.

[0038] The present application example provides a method for generating a reference image for an ultrasonic image by stripping out background, noise and other information in an ultrasonic image. First, an original ultrasonic image and pre-configured mask information are obtained. The mask information can process an area of ​​interest in the original ultrasonic image. One can be randomly selected from a mask information set as the pre-configured mask information, or a mask information can be randomly generated as the pre-configured mask information by configuring a generation rule of the mask information. Then, the original ultrasonic image and the mask information are multiplied. First, the mask information is changed to the same size as the original ultrasonic image, and then the original ultrasonic image and the mask information are multiplied to obtain a feature area ultrasonic image. The feature area ultrasonic image is input into a deep learning network, and features of the feature area ultrasonic image are extracted through the deep learning network. The extracted features are features that can affect the benchmark image, and then the extracted features are encoded and converted to obtain a first feature variable. Similarly, the original ultrasound image is input into a deep learning network to obtain a second feature variable. The shape features of the image are obtained according to the first feature variable, and the fine-grained features of the image are obtained according to the second feature variable. The shape features are low-dimensional features of the image, and the fine-grained features are high-dimensional features of the image. Then, a benchmark image is generated based on the shape features and the fine-grained features. The final generated benchmark image does not contain effective features, but only contains environmental features of background information and noise information in the original ultrasound image. It has the same environment as the original ultrasound image but does not provide effective information, and is close to an ideal benchmark image, thereby accurately representing information in the ultrasound image except for effective features, and providing an accurate benchmark image and accurate explanatory description for the prediction results of AI decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic flow chart of an embodiment of a method for generating a reference image of an ultrasound image of the present application;

[0040] Figure 2 A schematic diagram of a flow chart of an embodiment of calculating the increment of the original ultrasound image compared to the reference image in the present application;

[0041] Figure 3 A schematic diagram of the structure of an embodiment of a device for generating a reference image of an ultrasonic image of the present application;

[0042] Figure 4 This is a schematic block diagram of the structure of an embodiment of a computer device of the present application.

[0043] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0045] Reference Figure 1 An embodiment of the present application provides a method for generating a reference image of an ultrasound image. The method for generating a reference image of an ultrasound image includes steps S10-S60. The detailed description of each step of the method for generating a reference image of an ultrasound image is as follows.

[0046] S10. Obtain pre-configured mask information.

[0047] This embodiment is applied to the generation scenario of ultrasound reference images. The biggest obstacle to the penetration of artificial intelligence technology into the medical field is the "black box" problem of deep neural networks. Humans cannot believe the decisions made by unexplainable AI (Artificial Intelligence). Therefore, it is necessary to explain the prediction results made by AI. By calculating the contribution of different regions on the ultrasound image to the prediction results compared with the baseline image, the interpretability of the prediction results is determined. In order to accurately determine the contribution of different regions on the ultrasound image to the prediction results compared with the baseline image, this embodiment generates a corresponding reference image for each ultrasound image. The reference image is the information left after the original ultrasound image is stripped of effective features, that is, the environmental features in the original ultrasound image. First, obtain pre-configured mask information, the mask information is mask, and the mask information is composed of a×a small pieces. In one embodiment, the pre-configured mask information is randomly selected from the mask information set as the pre-configured mask information; in another embodiment, by configuring the generation rule of the mask information, a mask information is randomly generated as the pre-configured mask information.

[0048] S20: Acquire an original ultrasound image.

[0049] In this embodiment, after obtaining the preconfigured mask information, in order to generate a corresponding reference image for each ultrasound image, the original ultrasound image is obtained. Specifically, other medical devices can be connected. When the other medical devices capture the original ultrasound image, the original ultrasound image is obtained.

[0050] S30, performing product processing on the original ultrasonic image and the mask information to obtain an ultrasonic image of the characteristic region.

[0051] In this embodiment, after obtaining the pre-configured mask information and the original ultrasound image, the original ultrasound image is multiplied by the mask information, wherein the pixel value x on each small piece of the mask information is ij Follows a uniform distribution, and x ij ∈{0,1}, so that the mask information can process the area of ​​interest more accurately, and then change the mask information to the same size as the original ultrasound image, and then multiply the original ultrasound image and the mask information to obtain a feature area ultrasound image, and define the feature area ultrasound image as source A, and the original ultrasound image as source B.

[0052] S40. Input the characteristic region ultrasound image into a deep learning network to obtain a first characteristic variable; input the original ultrasound image into a deep learning network to obtain a second characteristic variable.

[0053] In this embodiment, after acquiring the original ultrasonic image and multiplying the original ultrasonic image with the mask information to obtain the characteristic area ultrasonic image, the characteristic area ultrasonic image is input into a deep learning network, and the deep learning network is used to extract features of the characteristic area ultrasonic image, the extracted features are features that can affect the baseline image, and the extracted features are encoded and converted to obtain a first feature variable; similarly, the original ultrasonic image is input into a deep learning network, and the deep learning network is used to extract features of the original ultrasonic image, the extracted features are features that can affect the baseline image, and the features extracted from the original ultrasonic image are encoded and converted to obtain a second feature variable.

[0054] S50: Obtain shape features of the image according to the first feature variables, and obtain fine-grained features of the image according to the second feature variables.

[0055] In this embodiment, after the ultrasonic image of the feature area is input into a deep learning network to obtain a first feature variable; after the original ultrasonic image is input into a deep learning network to obtain a second feature variable, a shape feature of the image is obtained according to the first feature variable, and a fine-grained feature of the image is obtained according to the second feature variable. The shape feature is a low-dimensional feature of the image, including a shape contained in the image and a position of the shape, and the fine-grained feature is a high-dimensional feature of the image, including a texture feature of the image and a color feature of the image. After the first feature variable and the second feature variable are obtained, they are combined as a style variable of the image, and the style variable y is defined as (y s ,y b ), where y S is the first characteristic variable, y bis the second feature variable. By combining the first feature variable with the second feature variable and inputting them into the convolution layer, the corresponding shape feature and fine-grained feature can be obtained.

[0056] S60: Generate a reference image according to the shape feature and the fine particle feature.

[0057] In this embodiment, after obtaining the shape feature of the image according to the first feature variable and obtaining the fine-grained feature of the image according to the second feature variable, a benchmark image is generated according to the shape feature and the fine-grained feature. The finally generated benchmark image does not contain effective features, but only contains environmental features of background information and noise information in the original ultrasound image. It has the same environment as the original ultrasound image but does not provide effective information, and is close to an ideal benchmark image. The background, noise and other information in the original ultrasound image are accurately stripped out to generate a benchmark image. The benchmark image can represent rich environmental features, thereby accurately representing information in the image other than effective features, and providing an accurate benchmark image and accurate explanatory description for the prediction results of AI decisions.

[0058] The present embodiment provides a method for generating a reference image for an ultrasonic image by stripping off information such as background and noise from an ultrasonic image. First, an original ultrasonic image and pre-configured mask information are obtained. The mask information can process an area of ​​interest in the original ultrasonic image. One can be randomly selected from a mask information set as pre-configured mask information, or a mask information can be randomly generated as pre-configured mask information by configuring a generation rule of mask information. Then, the original ultrasonic image and the mask information are multiplied. First, the mask information is changed to the same size as the original ultrasonic image, and then the original ultrasonic image and the mask information are multiplied to obtain a feature area ultrasonic image. The feature area ultrasonic image is input into a deep learning network, and features of the feature area ultrasonic image are extracted through the deep learning network. The extracted features are features that can affect the benchmark image, and then the extracted features are encoded and converted to obtain a first feature variable. Similarly, the original ultrasound image is input into a deep learning network to obtain a second feature variable. The shape features of the image are obtained according to the first feature variable, and the fine-grained features of the image are obtained according to the second feature variable. The shape features are low-dimensional features of the image, and the fine-grained features are high-dimensional features of the image. Then, a benchmark image is generated based on the shape features and the fine-grained features. The final generated benchmark image does not contain effective features, but only contains environmental features of background information and noise information in the original ultrasound image. It has the same environment as the original ultrasound image but does not provide effective information, and is close to an ideal benchmark image, thereby accurately representing information in the ultrasound image except for effective features, and providing an accurate benchmark image and accurate explanatory description for the prediction results of AI decisions.

[0059] In one embodiment, the step of inputting the characteristic region ultrasound image into a deep learning network to obtain a first characteristic variable; and inputting the original ultrasound image into a deep learning network to obtain a second characteristic variable comprises:

[0060] Inputting the characteristic region ultrasound image into a deep learning network, transforming the latent code of the characteristic region ultrasound image based on a nonlinear mapping network and an affine transformation in the deep learning network, and obtaining a first characteristic variable;

[0061] The original ultrasound image is input into a deep learning network, and the latent coding of the original ultrasound image is transformed based on a nonlinear mapping network and an affine transformation in the deep learning network to obtain a second feature variable.

[0062] In this embodiment, in the process of inputting the characteristic region ultrasound image into a deep learning network to obtain a first characteristic variable; and inputting the original ultrasound image into a deep learning network to obtain a second characteristic variable, the characteristic region ultrasound image is input into a deep learning network, the deep learning network includes a nonlinear mapping network, and then the latent coding of the characteristic region ultrasound image is transformed based on the nonlinear mapping network and affine transformation in the deep learning network to obtain a variable that can characterize the corresponding feature in the characteristic region ultrasound image, which is defined as the first characteristic variable; similarly, for the original ultrasound image, the original ultrasound image is input into a deep learning network, and the latent coding of the original ultrasound image is transformed based on the nonlinear mapping network and affine transformation in the deep learning network to obtain a variable that can characterize the corresponding feature in the original ultrasound image, which is defined as the second characteristic variable, thereby accurately extracting the image features of the original ultrasound image and the characteristic region ultrasound image, thereby improving the accuracy of reference image generation.

[0063] In one embodiment, obtaining the shape feature of the image according to the first feature variable and obtaining the fine-grained feature of the image according to the second feature variable includes:

[0064] Inputting the first feature variable into a low-resolution convolutional layer in a generative network to obtain shape features of the image;

[0065] The second feature variable is input into the high-resolution convolutional layer in the generative network to obtain fine-grained features of the image.

[0066] In this embodiment, in the process of obtaining the shape feature of the image according to the first feature variable and obtaining the fine-grained feature of the image according to the second feature variable, the first feature variable is input into a generative network, and the generative network includes multiple convolutional layers. The first feature variable is input into a low-resolution convolutional layer in the generative network, and the low-resolution convolutional layer can control the low-dimensional features of the generated image, that is, the shape feature of the image is obtained. At the same time, the second feature variable is input into a high-resolution convolutional layer in the generative network, and the high-resolution convolutional layer can control the high-dimensional features of the generated image, that is, the fine-grained feature of the image is obtained. By inputting different variables for controlling image generation into different convolutional layers, the high-dimensional features and low-dimensional features of the image are obtained, and the image features of the original ultrasound image and the characteristic area ultrasound image are accurately extracted, thereby improving the accuracy of baseline image generation.

[0067] In one embodiment, the low-resolution convolutional layer has a resolution of 4 2 -32 2 The high-resolution convolution layer has a resolution of 64 2 -1024 2 The convolutional layer.

[0068] In this embodiment, the low-resolution convolution layer has a resolution of 4 2 -32 2 The high-resolution convolution layer has a resolution of 64 2 -1024 2 By configuring multiple convolutional layers with different resolutions, different variables for controlling image generation are input into different convolutional layers, so as to obtain high-dimensional features and low-dimensional features of the image, and accurately extract image features of the original ultrasound image and the ultrasound image in the characteristic area, thereby improving the accuracy of benchmark image generation.

[0069] In one embodiment, after generating the reference image according to the shape feature and the fine-grain feature, the method further includes:

[0070] Inputting the original ultrasound image and the reference image into a classification network, and calculating the increment of the original ultrasound image compared to the reference image based on the classification network;

[0071] An attribution graph is generated based on the increments.

[0072] In this embodiment, after a reference image is generated according to the shape feature and the fine-grained feature, the original ultrasound image and the reference image are input into a classification network, the original ultrasound image is compared with the reference image, and an increment of the original ultrasound image compared to the reference image is calculated based on the classification network, the increment represents the contribution of each different region in the original ultrasound image to the determination result, and then an attribution map is generated according to the increment, the attribution map represents the contribution of each different region in the original ultrasound image to the determination result, and the attribution map can be used to determine whether the classification network correctly focuses on the correct image features and ignores the environmental features of the image, thereby improving the interpretability of the determination result.

[0073] In one embodiment, Figure 2 As shown, the inputting the original ultrasound image and the reference image into a classification network, and calculating the increment of the original ultrasound image compared to the reference image based on the classification network, comprises:

[0074] S61: Divide the original ultrasound image into a plurality of regional sub-images;

[0075] S62: Inputting each of the region sub-images and the reference image into a classification network respectively, and calculating, based on the classification network, a first contribution of each of the region sub-images to a prediction score of a respective result compared to the reference image;

[0076] S63: Acquire a second contribution of the environmental feature in the reference image;

[0077] S64: Determine the increment according to the first contribution and the second contribution.

[0078] In this embodiment, in the process of inputting the original ultrasonic image and the reference image into the classification network and calculating the increment of the original ultrasonic image compared with the reference image based on the classification network, the original ultrasonic image is first divided into a plurality of regional sub-images, for example, divided into 9*9 regional sub-images, and then each of the regional sub-images and the reference image are respectively input into the classification network, and the contribution of each of the regional sub-images to the prediction score of the respective results compared with the reference image is calculated based on the classification network, which is defined as a first contribution, and then a second contribution of the environmental features in the reference image is obtained, and then the increment is determined according to the first contribution and the second contribution. Specifically, the second contribution is subtracted from the first contribution to obtain the true contribution of each of the regional sub-images, and then the true contribution of each regional sub-image is coordinated to obtain the increment of the original ultrasonic image compared with the reference image, thereby accurately representing the contribution of different regions in the original ultrasonic image to the determination result and improving the interpretability of the determination result.

[0079] In one embodiment, after generating the attribution graph according to the increment, the method further includes:

[0080] Obtaining the target sub-region image with the highest contribution in the attribution map;

[0081] The effective feature information of the reference image is determined according to the target sub-region image.

[0082] In this embodiment, after generating an attribution map according to the increment, the area with the highest contribution in the attribution map is obtained and defined as a target sub-area image. Then, effective feature information of the baseline image is determined based on the target sub-area image, thereby determining the effective feature information of the baseline image. Manual comparison can be performed based on the effective feature information to improve the accuracy of the baseline image.

[0083] Reference Figure 3 The present application also provides a device for generating a reference image of an ultrasonic image, comprising:

[0084] Configuration information module 10, used to obtain pre-configured mask information;

[0085] The original image module 20 is used to obtain the original ultrasound image;

[0086] An image processing module 30, configured to perform product processing on the original ultrasonic image and the mask information to obtain an ultrasonic image of a characteristic region;

[0087] The variable conversion module 40 is used to input the characteristic region ultrasound image into the deep learning network to obtain a first characteristic variable; input the original ultrasound image into the deep learning network to obtain a second characteristic variable;

[0088] A feature extraction module 50, configured to obtain a shape feature of the image according to the first feature variable, and obtain a fine-grained feature of the image according to the second feature variable;

[0089] The reference image module 60 is used to generate a reference image according to the shape feature and the fine particle feature.

[0090] As described above, it can be understood that the various components of the device for generating a reference image of an ultrasonic image proposed in the present application can realize the functions of any one of the methods for generating a reference image of an ultrasonic image described above.

[0091] In one embodiment, the step of inputting the characteristic region ultrasound image into a deep learning network to obtain a first characteristic variable; and inputting the original ultrasound image into a deep learning network to obtain a second characteristic variable comprises:

[0092] Inputting the characteristic region ultrasound image into a deep learning network, transforming the latent code of the characteristic region ultrasound image based on a nonlinear mapping network and an affine transformation in the deep learning network, and obtaining a first characteristic variable;

[0093] The original ultrasound image is input into a deep learning network, and the latent coding of the original ultrasound image is transformed based on a nonlinear mapping network and an affine transformation in the deep learning network to obtain a second feature variable.

[0094] In one embodiment, obtaining the shape feature of the image according to the first feature variable and obtaining the fine-grained feature of the image according to the second feature variable includes:

[0095] Inputting the first feature variable into a low-resolution convolutional layer in a generative network to obtain shape features of the image;

[0096] The second feature variable is input into the high-resolution convolutional layer in the generative network to obtain fine-grained features of the image.

[0097] In one embodiment, the low-resolution convolutional layer has a resolution of 4 2 -32 2 The high-resolution convolution layer has a resolution of 64 2 -1024 2The convolutional layer.

[0098] In one embodiment, after generating the reference image according to the shape feature and the fine-grain feature, the method further includes:

[0099] Inputting the original ultrasound image and the reference image into a classification network, and calculating the increment of the original ultrasound image compared to the reference image based on the classification network;

[0100] An attribution graph is generated based on the increments.

[0101] In one embodiment, the step of inputting the original ultrasound image and the reference image into a classification network, and calculating the increment of the original ultrasound image compared to the reference image based on the classification network, comprises:

[0102] Dividing the original ultrasound image into a plurality of regional sub-images;

[0103] Inputting each of the region sub-images and the reference image into a classification network respectively, and calculating a first contribution of each of the region sub-images to a prediction score of a respective result compared to the reference image based on the classification network;

[0104] Acquire a second contribution of the environmental feature in the reference image;

[0105] The increment is determined according to the first contribution and the second contribution.

[0106] In one embodiment, after generating the attribution graph according to the increment, the method further includes:

[0107] Obtaining the target sub-region image with the highest contribution in the attribution map;

[0108] The effective feature information of the reference image is determined according to the target sub-region image.

[0109] Reference Figure 4 In the embodiment of the present application, a computer device is also provided. The computer device may be a mobile terminal, and its internal structure may be as follows: Figure 4As shown. The computer device includes a processor, a memory, a network interface, a display device and an input device connected through a system bus. Among them, the network interface of the computer device is used to communicate with an external terminal through a network connection. The display device of the computer device is used to display offline applications. The input device of the computer device is used to receive user input in offline applications. The processor designed by the computer is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium. The non-volatile storage medium stores an operating system, a computer program and a database. The database of the computer device is used to store raw data. When the computer program is executed by the processor, a method for generating a reference image of an ultrasound image is implemented.

[0110] The above-mentioned processor executes the above-mentioned method for generating a reference image of an ultrasonic image, which includes: obtaining preconfigured mask information; obtaining an original ultrasonic image; multiplying the original ultrasonic image with the mask information to obtain a feature area ultrasonic image; inputting the feature area ultrasonic image into a deep learning network to obtain a first feature variable; inputting the original ultrasonic image into a deep learning network to obtain a second feature variable; obtaining a shape feature of the image based on the first feature variable, and obtaining a fine-grained feature of the image based on the second feature variable; and generating a reference image based on the shape feature and the fine-grained feature.

[0111] The computer device provides a method for generating a reference image for an ultrasound image by stripping off information such as background and noise from an ultrasound image. First, an original ultrasound image and pre-configured mask information are obtained. The mask information can process an area of ​​interest in the original ultrasound image. One can be randomly selected from a mask information set as the pre-configured mask information, or a mask information can be randomly generated as the pre-configured mask information by configuring a generation rule of the mask information. Then, the original ultrasound image and the mask information are multiplied. First, the mask information is changed to the same size as the original ultrasound image. Then, the original ultrasound image and the mask information are multiplied to obtain a feature region ultrasound image. The feature region ultrasound image is input into a deep learning network. The deep learning network is used to extract features from the feature region ultrasound image. The extracted features are features that can affect the benchmark image, and then the extracted features are encoded and converted to obtain a first feature variable. Similarly, the original ultrasound image is input into a deep learning network to obtain a second feature variable. The shape features of the image are obtained according to the first feature variable, and the fine-grained features of the image are obtained according to the second feature variable. The shape features are low-dimensional features of the image, and the fine-grained features are high-dimensional features of the image. Then, a benchmark image is generated according to the shape features and the fine-grained features. The final generated benchmark image does not contain effective features, but only contains environmental features of background information and noise information in the original ultrasound image. It has the same environment as the original ultrasound image but does not provide effective information, and is close to an ideal benchmark image, thereby accurately representing information other than effective features in the ultrasound image, and providing an accurate benchmark image and accurate explanatory description for the prediction results of AI decisions.

[0112] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by the processor, a method for generating a reference image of an ultrasound image is implemented, comprising the steps of: obtaining preconfigured mask information; obtaining an original ultrasound image; multiplying the original ultrasound image with the mask information to obtain a feature area ultrasound image; inputting the feature area ultrasound image into a deep learning network to obtain a first feature variable; inputting the original ultrasound image into a deep learning network to obtain a second feature variable; obtaining a shape feature of the image based on the first feature variable, and obtaining a fine-grained feature of the image based on the second feature variable; and generating a reference image based on the shape feature and the fine-grained feature.

[0113] The computer-readable storage medium provides a method for generating a reference image for an ultrasonic image by stripping off information such as background and noise from an ultrasonic image. First, an original ultrasonic image and pre-configured mask information are obtained. The mask information can process an area of ​​interest in the original ultrasonic image. One can be randomly selected from a mask information set as pre-configured mask information, or a mask information can be randomly generated as pre-configured mask information by configuring a generation rule of mask information. Then, the original ultrasonic image is multiplied by the mask information. First, the mask information is changed to the same size as the original ultrasonic image. Then, the original ultrasonic image is multiplied by the mask information to obtain a characteristic region ultrasonic image. The characteristic region ultrasonic image is input into a deep learning network. The characteristic region ultrasonic image is processed by the deep learning network. Feature extraction, the extracted features are features that can affect the benchmark image, and then the extracted features are encoded and converted to obtain a first feature variable. Similarly, the original ultrasound image is input into a deep learning network to obtain a second feature variable. The shape features of the image are obtained according to the first feature variable, and the fine-grained features of the image are obtained according to the second feature variable. The shape features are low-dimensional features of the image, and the fine-grained features are high-dimensional features of the image. Then, a benchmark image is generated based on the shape features and the fine-grained features. The final generated benchmark image does not contain effective features, but only contains environmental features of background information and noise information in the original ultrasound image. It has the same environment as the original ultrasound image but does not provide effective information, which is close to an ideal benchmark image, thereby accurately representing information in the ultrasound image except for effective features, and providing an accurate benchmark image and accurate explanatory description for the prediction results of AI decisions.

[0114] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided in this application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0115] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, device, article or method including the element.

[0116] The above description is only a preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for generating a reference image of an ultrasonic image, characterized in that: include: Get pre-configured mask information; Acquire original ultrasound images; Performing a product process on the original ultrasonic image and the mask information to obtain an ultrasonic image of the characteristic region; Inputting the characteristic region ultrasound image into a deep learning network to obtain a first characteristic variable; Inputting the original ultrasound image into a deep learning network to obtain a second feature variable; Obtaining a shape feature of the image according to the first feature variable, and obtaining a fine-grained feature of the image according to the second feature variable; generating a reference image according to the shape feature and the fine-grain feature; After generating the reference image according to the shape feature and the fine-grain feature, the method further includes: Inputting the original ultrasound image and the reference image into a classification network, and calculating the increment of the original ultrasound image compared to the reference image based on the classification network; generating an attribution graph based on the increments; The step of inputting the original ultrasound image and the reference image into a classification network, and calculating the increment of the original ultrasound image compared to the reference image based on the classification network, comprises: Dividing the original ultrasound image into a plurality of regional sub-images; Inputting each of the regional sub-images and the reference image into a classification network respectively, and calculating a first contribution of each of the regional sub-images to a prediction score of a classification result compared to the reference image based on the classification network; Acquire a second contribution of the environmental feature in the reference image; determining the increment according to the first contribution and the second contribution; After generating the attribution graph according to the increment, the method further includes: Obtaining the target sub-region image with the highest contribution in the attribution map; The effective feature information of the reference image is determined according to the target sub-region image.

2. The method for generating a reference image of an ultrasonic image according to claim 1, characterized in that: The ultrasonic image of the characteristic region is input into a deep learning network to obtain a first characteristic variable; The original ultrasound image is input into a deep learning network to obtain a second feature variable, including: Inputting the characteristic region ultrasound image into a deep learning network, transforming the latent code of the characteristic region ultrasound image based on a nonlinear mapping network and an affine transformation in the deep learning network, and obtaining a first characteristic variable; The original ultrasound image is input into a deep learning network, and the latent coding of the original ultrasound image is transformed based on a nonlinear mapping network and an affine transformation in the deep learning network to obtain a second feature variable.

3. The method for generating a reference image of an ultrasonic image according to claim 1, characterized in that: The step of obtaining the shape feature of the image according to the first feature variable and obtaining the fine-grained feature of the image according to the second feature variable includes: Inputting the first feature variable into a low-resolution convolutional layer in a generative network to obtain shape features of the image; The second feature variable is input into the high-resolution convolutional layer in the generative network to obtain fine-grained features of the image.

4. The method for generating a reference image of an ultrasonic image according to claim 3, characterized in that: The low-resolution convolution layer has a resolution of 4 2 -32 2 The high-resolution convolution layer has a resolution of 64 2 -1024 2 The convolutional layer.

5. A device for generating a reference image of an ultrasonic image, used to execute the method according to any one of claims 1 to 4, characterized in that: include: A configuration information module, used to obtain pre-configured mask information; An original image module, used for acquiring original ultrasound images; An image processing module, used for performing product processing on the original ultrasonic image and the mask information to obtain an ultrasonic image of a characteristic region; A variable conversion module, used for inputting the characteristic region ultrasound image into a deep learning network to obtain a first characteristic variable; Inputting the original ultrasound image into a deep learning network to obtain a second feature variable; A feature extraction module, used for obtaining a shape feature of the image according to the first feature variable, and obtaining a fine-grained feature of the image according to the second feature variable; A reference image module is used to generate a reference image according to the shape feature and the fine particle feature.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the processor implements the steps of the method for generating a reference image of an ultrasound image according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for generating a reference image of an ultrasound image according to any one of claims 1 to 4 are implemented.

Images