An image generation method based on a generative model, a medium and an apparatus

Abstract

The present application relates to the technical field of image generation, and in particular to an image generation method based on a generative model, a medium and equipment, a multi-channel conditional input tensor graph is constructed through disease type and spatial attribute information, different disease types are mapped according to different channels, non-zero pixel marks disease areas, so that the target image generation model can learn and generate disease characteristics by channel, and mutual interference of multi-disease characteristics is avoided; through a spatial self-adaptive normalization module built in the generator, spatial difference modulation parameters are predicted by using the multi-channel conditional input tensor graph, so that the model can inject disease types and spatial attribute information into feature maps at each intermediate generation stage, and disease and non-disease areas are processed in a targeted manner, the relative characteristic relationship of the disease area is retained, high-frequency texture details are strengthened, and the local characteristics of the target disease image are enriched, so that high-quality image generation can be controlled according to user-specified attributes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image generation, in particular to an image generation method based on a generative model, a medium and an equipment. BACKGROUND

[0002] Ground Penetrating Radar (GPR) is widely used in underground disease detection of airport runways and other scenes due to its advantages of non-destructive, fast and comprehensive coverage. This technology transmits high-frequency electromagnetic waves and receives their reflected signals, detects underground targets according to propagation characteristics and time delay, and obtains B-scan disease images.

[0003] Traditionally, GPR image data is usually obtained by searching the runway used in actual operation of the airport with ground penetrating radar. However, the overall detection process of the runway with ground penetrating radar has high time and space cost requirements, and the conventional detection cycle is long in the airport operation, and the image data itself has high privacy, making it difficult to obtain a large number of real detection images for research.

[0004] Deep learning, as a more efficient and objective means, can generate high-quality images close to reality and meeting various disease attribute requirements by learning the characteristics of real disease images. In recent years, with the development of deep learning, generative adversarial networks, variational autoencoders and conditional generative networks have been widely introduced into the GPR B-scan disease image generation task. This kind of method can directly learn the latent distribution from real data and automatically capture complex features, so as to generate samples closer to real GPR images. However, due to the significant differences in geological environment of different airports, including the composition, structure and humidity of underground medium. At the same time, underground diseases have irregularity and diversity in shape and size. These factors together increase the complexity of disease features in image data, which is specifically manifested as: large intra-class feature difference. The same type of disease may exhibit significantly different reflection characteristics in different environments; and high inter-class feature similarity, different categories of targets may exhibit similar image features in some cases, which can easily lead to model mis-detection.

[0005] Correspondingly, the existing methods still have obvious deficiencies in the process of learning and expressing disease features. Specifically, the current technology often has difficulty in fully capturing and distinguishing the independent features of different types of diseases in the generation stage, resulting in that the morphological features of each type of disease in the generated results are not clear and accurate enough. At the same time, when multiple diseases need to be presented in one B-scan image, the existing methods often have difficulty in effectively fusing and coexisting multiple disease features, and some disease features may be weakened or even lost. The lack of disease feature learning and complex expression ability directly limits the expressiveness and application value of the generated image in real scenarios.

[0006] Therefore, how to more comprehensively understand the manifestation forms of various diseases and realize fine control on local features in the image generation operation process so as to improve the image generation quality becomes a problem to be solved. SUMMARY

[0007] In order to solve the above technical problems, the present application adopts the technical scheme of an image generation method based on a generative model, which comprises the following steps:

[0008] S1, according to the one-to-one correspondence relationship between the preset disease types and the channel numbers, determining the target channel corresponding to each target disease type in the target disease attribute information, wherein the preset disease types at least include void, loose and crack.

[0009] S2, according to the spatial attribute information corresponding to each target disease type in the target disease attribute information, locating the disease area of the target channel corresponding to each target disease type in the preset multi-channel tensor graph, wherein the preset multi-channel tensor graph is all zero value, the number of channels of the preset multi-channel tensor graph is consistent with the total number of preset disease types, and the spatial resolution of the preset multi-channel tensor graph is the same as the spatial resolution of the target disease image to be generated.

[0010] S3, updating the pixel values in all disease areas in the preset multi-channel tensor graph to non-zero identification values to obtain a multi-channel conditional input tensor graph.

[0011] S4, processing the multi-channel conditional input tensor graph according to the target image generation model to obtain the target disease image, wherein the spatial adaptive normalization module is built in the target generator of the target image generation model, and the spatial adaptive normalization module is used to spatially adaptively modulate the feature map extracted by the target generator according to the disease area in the multi-channel conditional input tensor graph in the image generation process.

[0012] The present application also provides a non-transitory computer readable storage medium, which stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by a processor to implement the above-mentioned image generation method based on a generative model.

[0013] The present application also provides an electronic device comprising a processor and the above-mentioned non-transitory computer readable storage medium.

[0014] The present application has at least the following beneficial effects: by constructing a multi-channel condition input tensor graph, the disease type and spatial attribute are converted into structured condition information, different disease types and non-zero pixel markers are mapped to disease areas through different channels, so that the target image generation model can learn and generate corresponding disease characteristics in a targeted manner, avoid mutual interference and confusion of multi-disease characteristics in the modulation process, and make the generation process realize accurate and controllable disease attribute; by using the spatial adaptive normalization module built in the generator, the spatial difference modulation parameters are directly predicted by the multi-channel condition input tensor graph in the feature map generation process, so that the target image generation model can inject disease type and spatial attribute information into the feature map at each intermediate generation stage, and process the disease area and non-disease area in a targeted manner, both retaining the relative feature relationship of the disease area and strengthening the high-frequency texture details, avoiding the mode collapse and detail loss problem of the traditional image generation model under small sample, so that the generated target disease image has high real local features, thereby realizing controllable high-quality image generation according to user-specified attributes based on a small amount of real samples. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0016] Figure 1 A flowchart of an image generation method based on a generative model provided by the first embodiment of the present application. DETAILED DESCRIPTION

[0017] The technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0018] It should be noted that the terms first, second, etc. in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It can be understood that the above-described terms for distinguishing similar objects can be interchanged under appropriate circumstances, so that the present application can also be implemented in other embodiments in addition to the above-described embodiment or described embodiment. In addition, the terms include and have and any variation are intended to cover non-exclusive inclusion, for example, a process, method, system, product or server including a series of steps or units does not have to be limited to the clearly listed steps or units, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0019] Embodiment one

[0020] The embodiment one provides a generative model-based image generation method, as shown in the figure, the generative model-based image generation method comprises the following steps: Figure 1

[0021] S1, according to the one-to-one correspondence between the preset disease type and the channel number, determine the target channel corresponding to each target disease type in the target disease attribute information, wherein the preset disease type at least includes empty, loose and crack.

[0022] Wherein, the preset disease type is a pre-defined set of underground disease categories that need to be generated, at least containing empty (cavity formed by separation of underground structure layers), loose (loose area formed by uneven underground medium density), crack (fracture gap of underground structure), which can be extended to other disease types, used to define the generation range of the image generation model, and provide basic classification basis for channel mapping. For example, the runway is the core part of the airport infrastructure, which bears the key task of aircraft take-off and landing. With the increase of aircraft flow and the long-term high load use of the runway, various diseases such as empty, loose and crack are prone to occur in the underground structure below the runway, which not only affects the carrying capacity of the runway, but also may cause serious safety accidents.

[0023] The channel number is the unique identifier of each channel in the multi-channel tensor graph (such as 0, 1, 2, …), which is assigned in a continuous increasing manner, and is used as a digital index of disease type, so that the image generation model can quickly locate the disease characteristics of the corresponding channel.

[0024] A disease type is assigned only one channel number, and one channel number corresponds to only one disease type, without intersection and redundancy, avoiding confusion of different disease characteristics in the same channel, and ensuring the purity of single-channel disease characteristics.

[0025] ​The target disease attribute information is a user-specified disease-related requirement to be generated, including a target disease type to be generated, and spatial attribute information such as spatial position information, size information, and boundary range information of the associated disease.

[0026] The target channel is a specific channel determined through a mapping relationship with the target disease type, and is used for subsequent positioning and marking of a disease area in the multi-channel tensor graph, so as to delimit an exclusive generation space of a disease feature and ensure independent processing of different disease features in different channels.

[0027] Through the unique mapping rule of the preset disease type and the channel number, the target disease type of the user requirement is corresponded to a specific channel of the multi-channel tensor graph, so that the abstract disease type requirement is converted into channel index information recognizable by the image generation model, precise control of the disease type in the generation process is realized, and a logical basis is provided for subsequent positioning of the disease area and precise generation of the disease feature in different channels.

[0028] In a specific embodiment, S1 further includes the following steps:

[0029] S111, acquiring a plurality of real underground disease images under multiple scenes, and real disease attribute information corresponding to each real underground disease image, wherein the real disease attribute information includes a plurality of real disease types and real spatial attribute information corresponding to the real underground disease image.

[0030] S112, deduplicating all real disease types in all real disease attribute information to obtain N preset disease types after deduplication, wherein N≥3.

[0031] S113, assigning an independent and continuous channel number to each preset disease type, wherein each channel number uniquely corresponds to a channel in the preset multi-channel tensor graph.

[0032] The real underground disease images under multiple scenes are B-scan images of ground penetrating radar collected from different airport runways, different geological environments (such as dry soil, wet soil, and different rock layers), and different disease development stages, and cover the real performance of typical diseases such as void, loose, and crack, so as to provide real disease feature samples and ensure that the preset disease types conform to the actual application scenarios and avoid generating disease features that are divorced from reality.

[0033] The real disease attribute information is annotation data corresponding one-to-one to the real underground disease image, and includes real disease types and real spatial attribute information (such as the upper left corner, lower right corner coordinates, width, and height of the disease in the image), which provides an original basis for extracting the preset disease types and a spatial reference standard for subsequent positioning of the disease area.

[0034] Specifically, in the airport runway scenario, robots equipped with GPR (Gas Precision Image Processing) devices collected real underground defect image data on 11 airport runways. A 14-channel Raptor™ GPR with equal spacing (0.7 meters between each channel) was used, equipped with a 900MHz antenna and set to distance-triggered mode to ensure synchronous sampling. During image data acquisition, the robot performed a full-coverage scan of the airport runways according to a pre-planned path, ensuring that every area of ​​the runway was detected without omission. The collected GPR data was transmitted in real-time via 4G or Wi-Fi to a nearby mobile data analysis center for storage and preprocessing. The preprocessing removed some noise and background interference, enhanced the target signal, thereby improving data quality and the accuracy of subsequent analysis, forming an airport runway underground dataset as a real underground image sample for the generation method research. Furthermore, image samples containing different defect types were selected from the real airport runway underground dataset and its corresponding defect attribute information. Disease attribute information typically includes disease type information, its detection box coordinates in the image, and the width and height of the disease area. By writing automated scripts, disease attribute information can be extracted, organized, and stored in a standardized table format, serving as an important basic data source for subsequent conditional tensor generation and model training.

[0035] After extracting real disease types from all real disease attribute information and removing duplicates, a standardized disease classification set is formed, resulting in N preset disease types after deduplication, where N≥3, including at least voids, looseness, and cracks. This is used to clarify the generation scope of the image generation model, ensuring that all generated disease types are supported by real data and avoiding meaningless virtual disease types.

[0036] Based on the principle that one disease type corresponds to one channel number, a unique numerical identifier (such as 0, 1, 2...N-1) is assigned to each preset disease type according to an ordered list of preset disease types (such as [void, loose, crack]). The numbering is continuous without intervals or repetitions, thereby establishing a one-to-one index between disease type and multi-channel tensor image, enabling the image generation model to quickly locate the corresponding channel and avoid confusion between different diseases in the same channel.

[0037] The channels of the preset multi-channel tensor map are independent data layers in the multi-channel tensor map used to carry the features of a single disease type. Each channel corresponds to only one preset disease type, providing a dedicated feature generation space for each disease type, ensuring the purity of disease features within a single channel, and preventing mutual interference during subsequent fusion.

[0038] The above, through multi-scene real data collection, the preset disease type covers the geological difference and disease characteristics of different scenes, avoids the generated disease image only adapting to a single scene, and improves the cross-scene generalization ability of the model generation result; through independent and continuous channel number allocation, each disease type has a dedicated channel, avoids confusion of different disease characteristics in the same channel, provides a clear carrier for subsequent channel positioning of the disease area, and guarantees the accuracy of disease image generation.

[0039] S2, according to the spatial attribute information corresponding to each target disease type in the target disease attribute information, in the preset multi-channel tensor graph, positioning the disease area of the target channel corresponding to each target disease type, wherein the preset multi-channel tensor graph is all zero value, the number of channels of the preset multi-channel tensor graph and the total number of preset disease types are consistent, and the spatial resolution of the preset multi-channel tensor graph and the spatial resolution of the target disease image to be generated are the same.

[0040] In a specific embodiment, S2 includes the following steps:

[0041] S21, according to the total number of preset disease types and the spatial resolution of the target disease image to be generated, initializing a preset multi-channel tensor graph with all zero values.

[0042] S22, for any target disease type, according to the spatial attribute information of the current target disease type, the coordinates (x1, y1, x2, y2) of the disease area of the current target disease type in the corresponding target channel are calculated, wherein the disease area is a rectangular area, (x1, y1) is the upper left corner coordinate of the disease area, and (x2, y2) is the lower right corner coordinate of the disease area.

[0043] S23, traversing all target disease types, positioning the disease area of each target disease type corresponding to the target channel.

[0044] The spatial attribute information includes spatial position information (such as upper left corner pixel point (x1, y1), lower right corner pixel point (x2, y2)), size information (such as width W=x2-x1, height H=y2-y1), and boundary range information (such as the four edges of the rectangular area are upper boundary y=y1, lower boundary y=y2, left boundary x=x1, and right boundary x=x2), and can also include shape information (such as rectangle, irregular polygon), which is used to clearly define the spatial range of the disease in the tensor graph and is the direct basis for positioning the disease area.

[0045] The preset multi-channel tensor map is a pre-constructed all-zero four-dimensional tensor, and the format is usually (B, C, H, W), wherein B is a batch size, C is a channel number, H is a height, and W is a width. The channel number C is equal to a total number N of preset disease types. A spatial resolution (HxW) is consistent with a target disease image to be generated, so as to ensure that a disease area positioning result in the tensor map is completely matched with a disease size of the target disease image to be generated, and to avoid position deviation and size distortion caused by resolution difference. The all-zero state reserves space for subsequent marking of the disease area, and the multi-channel structure is suitable for sub-channel positioning requirements.

[0046] The disease area of the target channel is a specific pixel area in the target channel corresponding to the target disease type in the preset multi-channel tensor map, which is delimited by the spatial attribute information, and is used to lock a dedicated generation area for each target disease type, so as to ensure that only corresponding disease characteristics are generated in the area subsequently, and to avoid cross-area and cross-channel interference.

[0047] The above, by analyzing and verifying the spatial attribute information, the disease area parameters positioned are consistent with the tensor map resolution requirements, positioning errors caused by invalid spatial parameters are avoided, and the disease position and size generated subsequently are consistent with user requirements; by initializing the all-zero preset multi-channel tensor map, an independent channel carrier is provided for each preset disease type, and the all-zero state facilitates subsequent clear marking of the disease area by using non-zero identification values, and avoids interference of initial data on the positioning result; by positioning the disease area in sub-channels, each target disease type is delimited in the corresponding target channel, a one-to-one correspondence between the disease type, the channel and the disease area is achieved, different diseases are prevented from competing for generation space in the same channel, and the independence of subsequent disease characteristic generation is ensured; by setting the resolution of the preset multi-channel tensor map to be consistent with that of the target disease image to be generated, the disease area size positioned in the tensor map can be directly mapped to the disease size of the finally generated image, the positioning area and the generation result are prevented from being mismatched due to resolution difference, and the spatial accuracy of disease generation is improved.

[0048] S3, updating pixel values in all disease areas in the preset multi-channel tensor map to non-zero identification values to obtain a multi-channel conditional input tensor map.

[0049] In a specific embodiment, S3 includes the following steps:

[0050] S31, updating pixel values in all disease areas in the preset multi-channel tensor map to non-zero identification values, wherein the non-zero identification values are preset fixed values.

[0051] S32, keeping pixel values in non-disease areas in the preset multi-channel tensor map as zero values to obtain the multi-channel conditional input tensor map.

[0052] The non-zero identification value is a fixed non-zero value preset for marking the disease area, needs to match the tensor graph data type, and does not change with the disease type and channel, such as being set to 1 when based on the PyTorch framework, which matches the tensor graph torch.float32 data type; if it is an 8-bit image format, it can be set to 255. The non-zero identification value is used as a numerical label of the disease area, so that the image generation model can quickly distinguish the area that needs to generate the disease from the area that does not need to generate, and has uniqueness in the full tensor graph, that is, only the disease area is used, to avoid confusion in area identification.

[0053] In the preset multi-channel tensor graph, the target channel (such as the emptying corresponding channel 0) is first selected, and then the rectangular disease area in the channel is locked according to the coordinates (x1, y1, x2, y2), and all pixel values in the disease area are updated to the non-zero identification value through a tensor index operation.

[0054] The non-disease area is all pixel areas in the preset multi-channel tensor graph except the positioned disease area, including the blank area without disease and the non-target area of other channels, which is used to clearly define the range of the image generation model prohibited to generate disease characteristics, to avoid the model from generating diseases in the non-target area by keeping zero value, and to ensure the accuracy of the generated results.

[0055] The above, by marking with a unified non-zero identification value, the numerical signals of all disease areas are consistent, the image generation model does not need to analyze multiple numerical values, and the target area can be quickly identified, the signal processing complexity of the image generation model is reduced, and the feature modulation efficiency is improved; through the binary processing of disease area updating and non-disease area zero value keeping, the signal contrast of the multi-channel conditional input tensor graph is extremely high, the area misjudgment caused by fuzzy numerical value is avoided, and the identification accuracy of the image generation model for the disease position is ensured.

[0056] S4, processing the multi-channel conditional input tensor graph according to the target image generation model to obtain a target disease image, wherein the target generator of the target image generation model has a built-in spatial adaptive normalization module, which is used to spatially adaptively modulate the feature map extracted by the target generator according to the disease area in the multi-channel conditional input tensor graph during the image generation process.

[0057] The target image generation model is a deep learning system for converting a multi-channel conditional input tensor map to a target disease image. It is based on the SPADE-GAN (Spatially-Adaptive Normalization Generative Adversarial Network) architecture, and consists of a generator and a discriminator. In the training phase, the generator and the discriminator work together, while in the inference phase, only the generator is needed. In this embodiment, the generator is specifically designed for generating B-scan ground disease images using ground penetrating radar, and meets the requirements of few-sample learning and precise spatial constraints.

[0058] The generator is the image manufacturing core of the target image generation model. It adopts a progressive architecture with noise input, feature construction, adaptive modulation, resolution enhancement, and image output. It has built-in fully connected layers, up-sampling residual blocks (including spatial adaptive normalization modules), convolution layers, and activation functions. The core objective is to fuse random noise and conditional constraints to generate target disease images.

[0059] Traditional normalization modules use the same normalization parameters for the entire feature map, which cannot distinguish between disease areas and non-disease areas, leading to disease feature diffusion or loss. The spatial adaptive normalization module (SPADE) breaks through the limitations of traditional normalization (such as Batch Norm) and realizes spatial differentiation modulation based on a multi-channel conditional input tensor map, ensuring that disease features are only generated in designated areas and achieving precise conditional constraints.

[0060] In a specific embodiment, S4 includes the following steps:

[0061] S41, input a random noise vector into the fully connected layer in the target generator through the noise input port, and perform linear mapping on the random noise vector based on the fully connected layer to obtain a one-dimensional initial feature vector.

[0062] S42, adjust the spatial resolution of the one-dimensional initial feature vector through Reshape to obtain an initial feature map, wherein the spatial resolution of the initial feature map is less than the spatial resolution of the multi-channel conditional input tensor map.

[0063] S43, pre-process the multi-channel conditional input tensor map, and input the pre-processed multi-channel conditional input tensor map into the target generator through the conditional input port and the M up-sampling residual blocks in series, wherein M is an integer greater than 2, and the pre-processing includes resolution adjustment, tensor conversion, and normalization operation.

[0064] S44, input the initial feature map to the first upsampling residual block, and sequentially pass through the modulation and upsampling of the M upsampling residual blocks to obtain a reference feature map output by the Mth upsampling residual block, wherein the spatial resolution of the reference feature map is the same as that of the multi-channel conditional input tensor map.

[0065] S45, convolve and map the reference feature map according to the convolution layer and the activation function in the target generator to obtain a target disease image.

[0066] wherein the random noise vector is a random numerical vector with fixed dimensions (such as 128 dimensions) and elements following a normal distribution (such as [-1, 1] range), having randomness and diversity, providing a diversity basis for image generation, so that different target disease images with different details (such as crack edge texture differences) can be generated under the same conditions.

[0067] The fully connected layer is a network layer in the target generator responsible for linear mapping, which converts the low-dimensional noise vector into a high-dimensional feature vector through a weight matrix, realizing the conversion from noise without spatial structure to high-dimensional features, and providing data support for subsequent construction of feature maps. Specifically, the shorthand noise vector is input to the fully connected layer through the noise input port of the target generator, and the fully connected layer converts the random noise vector into a one-dimensional initial feature vector through linear operation of “noise vector x weight matrix + bias”, completing the conversion of “low-dimensional random features to high-dimensional features”.

[0068] According to the subsequent upsampling requirements, the dimensions of the initial feature map are set to four dimensions (batch dimension B, channel dimension C, height H, width W), such as (1, 1024, 8, 8), i.e. B=1, C=1024, H=8, W=8. The Reshape operation is called to convert the one-dimensional initial feature vector (1x65536) into a four-dimensional initial feature map, ensuring that the total number of elements remains unchanged (1x1024x8x8=65536). The initial feature map has channel, height, and width spatial properties, and its resolution is lower than that of the multi-channel conditional input tensor map, serving as the initial input for the subsequent M upsampling residual blocks, reserving space for the gradual resolution improvement of the M upsampling residual blocks.

[0069] The preprocessing includes resolution adjustment, tensor conversion, and normalization operation. For example, the resolution adjustment is to unify the resolution of the multi-channel conditional input tensor map to the resolution of the target disease image to be generated (such as 256x256) through bilinear interpolation, ensuring that the size matches the final generated result. The tensor conversion is to convert the multi-channel conditional input tensor map from a PIL image and a numpy array to a four-dimensional tensor (such as (1, 3, 256, 256)). The normalization is to normalize the pixel values of the multi-channel conditional input tensor map to the range of [0, 1] to adapt to the input requirements of the image generation model.

[0070] The up-sampling residual block (M in series) is a residual network unit containing a SPADE module and an up-sampling operation, M is an integer greater than 2 (such as 6), which realizes the dual functions of feature modulation and resolution improvement. Among them, the SPADE module reads the multi-channel conditional input tensor graph, predicts the modulation parameters and acts on the feature map, realizes the spatial precise constraint: only in the disease area marked by the condition graph, the feature generation is strengthened, and in the non-disease area, the generation is inhibited, which guarantees the accuracy of disease location and type. After being processed by M up-sampling residual blocks, the output reference feature map with the same resolution as the multi-channel conditional input tensor Figure 1 carries complete disease characteristics and spatial information (such as 256x256).

[0071] The reference feature map is input into the 1x1 convolution layer of the target generator, and the channel number is mapped from the current dimension (such as 512) to 1, which is suitable for the single-channel gray image features of the ground penetrating radar B-scan image. Through the Tanh activation function, the pixel value of the convolution feature map is mapped to the range of [-1, 1], and the initial generated image is obtained; and through pixel value inverse mapping, according to the formula pixel=(pixel+1) / 2, pixel is the pixel value, the pixel value is converted from [-1, 1] to [0, 1] range, and the tensor format image is further converted into a PIL image or a numpy array. If storage is required, it is further converted into an 8-bit grayscale image (0-255), and the final target disease image is obtained.

[0072] Based on the diversity provided by the random noise vector, the initial feature skeleton is constructed through the full connection layer and the reshape, and then the multi-channel conditional input tensor graph is used as the spatial constraint, and the up-sampling residual block containing the spatial adaptive normalization module is used to realize the feature modulation and resolution improvement, and finally the target disease image meeting the requirements is output through the convolution and the activation function, which realizes the combination of random diversity and precise spatial constraint, so that the target image generation model generates specific disease characteristics in the specified area, and guarantees the authenticity and detail richness of the target disease image.

[0073] In a specific embodiment, each up-sampling residual block contains a spatial adaptive normalization module, and S44 includes the following steps:

[0074] S441, in the ith up-sampling residual block, the multi-channel conditional input tensor graph is read according to the adaptive normalization module, and the multi-channel conditional input tensor graph is down-sampled by interpolation method to obtain the ith reference tensor graph corresponding to the ith up-sampling residual block, wherein the spatial resolution of the ith reference tensor graph is consistent with the spatial resolution of the input feature map corresponding to the ith up-sampling residual block, i = 1, 2, …, M, when i = 1, the input feature map corresponding to the first up-sampling residual block is the initial feature map, and when i ≥ 2, the input feature map corresponding to the ith up-sampling residual block is the reference feature map corresponding to the (i-1)th up-sampling residual block.

[0075] S442, the disease area of each channel in the ith reference tensor graph is identified according to the adaptive normalization module.

[0076] S443, according to the spatial distribution of the disease area read, the modulation parameter of each spatial position in the input feature map corresponding to the ith up-sampling residual block is predicted, wherein the modulation parameter includes a scaling coefficient and an offset coefficient.

[0077] S444, the input feature map corresponding to the ith up-sampling residual block is normalized and spatially adaptively modulated by the modulation parameter to obtain the modulated feature map corresponding to the ith up-sampling residual block.

[0078] S445, the modulated feature map corresponding to the ith up-sampling residual block is up-sampled by interpolation method to obtain the reference feature map corresponding to the ith up-sampling residual block.

[0079] S446, all up-sampling residual blocks are traversed in the order of i = 1, 2, …, M to obtain the reference feature map output by the Mth up-sampling residual block.

[0080] Wherein, the resolution of the input feature map corresponding to the ith up-sampling residual block is 8x8 when i = 1, 16x16 when i = 2, and so on. The SPADE module reads the preprocessed multi-channel conditional input tensor graph (such as 256x256), and uses bilinear interpolation method to down-sample it to adjust the resolution to be consistent with the spatial resolution of the input feature map corresponding to the ith up-sampling residual block. Figure 1 (For example, 256x256→8x8), to obtain the ith reference tensor graph.

[0081] The SPADE module traverses each channel (e.g., channel 0 corresponds to void, channel 1 corresponds to loose) of the i-th reference tensor map based on the rule of "non-zero value=disease area, zero value=non-disease area", and for each channel, marks the spatial positions of all pixel values as non-zero identification values to form a disease area mask exclusive to the channel (e.g., in the mask of channel 0, the (x1, y1)-(x2, y2) area is 1, and the rest is 0); the disease area masks of all channels are summarized to form a "channel-area" correspondence table, which clearly shows the spatial distribution of each disease type in the i-th reference tensor map (e.g., channel 0 has a void area, and channel 2 has a crack area).

[0082] The SPADE module has a small convolutional network (e.g., 2 layers of 3x3 convolution, with the output channel number being twice the input feature map channel number) built-in, which inputs the i-th reference tensor map into the convolutional network and outputs two tensors with the same size as the input feature map based on the disease area distribution of the reference tensor map, i.e., scaling coefficient γ and offset coefficient β (e.g., if the input feature map is (1, 1024, 8, 8), then γ and β are both (1, 1024, 8, 8)), ensuring that the γ / β values corresponding to the disease area have the property of strengthening features (e.g., γ is slightly greater than 1, and β is close to the true disease feature mean), and the γ / β values corresponding to the non-disease area have the property of suppressing features (e.g., γ is close to 0, and β is close to 0). Among them, the scaling coefficient γ controls the scaling of feature intensity, and the offset coefficient β controls the offset of feature value, realizing spatial differentiation modulation: in the disease area, the features are strengthened by γ / β, and in the non-disease area, the features are suppressed to avoid the spread of disease features.

[0083] Further, instance normalization is performed on the input feature map of the i-th up-sampling residual block to eliminate the brightness and contrast differences of different channel features, and the formula is: norm(x)=(x-mean) / std, where norm(x) is the output result of performing instance normalization on the input feature map x of the i-th up-sampling residual block, mean is the mean of the pixel values of all pixels in the corresponding channel, and std is the standard deviation of the pixel values of all pixels in the corresponding channel.

[0084] The normalized feature map and γ, β are calculated according to the formula modulated_x=norm(x)×(1+γ)+β to realize spatial adaptive modulation, i.e., the features in the disease area are strengthened (e.g., the hyperbolic feature in the void area is more obvious), and the features in the non-disease area are suppressed. After modulation, the modulated feature map modulated_x with the same resolution and channel number as the input feature map is obtained (e.g., (1, 1024, 8, 8)), and its feature distribution has adapted to the disease area constraint of the conditional map, carrying the disease features after precise constraint, providing a high-quality feature basis for subsequent up-sampling, and avoiding the introduction of invalid information when the resolution is improved.

[0085] The bilinear interpolation method is used to perform 2 times upsampling on the modulated feature map (e.g., 8x8→16x16, 16x16→32x32), to ensure that the spatial resolution of the upsampled feature map is 2 times the input. In the bilinear interpolation process, the disease details (e.g., linear features of crack edges, boundary contours of void areas) of the modulated feature map are preserved through weight distribution, to avoid blurring of the details. After upsampling is completed, the reference feature map corresponding to the i-th upsampled residual block (e.g., (1, 1024, 16, 16) when i=1) is obtained, and the number of channels is the same as that of the modulated feature map. Figure 1 .

[0086] In order of i=1→2→…→M, the reference feature map of the i-th upsampled residual block is taken as the input feature map of the i+1-th upsampled residual block (e.g., the 16x16 reference feature map of i=1 is taken as the input of i=2), and the steps of S441-S445 are repeated to accumulate disease details. After completing the processing of each upsampled residual block, the resolution of the reference feature map is recorded (e.g., 8x8→16x16→32x32→…→256x256), to ensure that the resolution is increased by 2 times, and finally matches the multi-channel conditional input tensor. Figure 1 When i=M, the reference feature map (e.g., 256x256) output by the M-th upsampled residual block is the final reference feature map.

[0087] As described above, the reference tensor map down-sampling alignment ensures that the conditional constraints of each upsampled residual block are completely matched with the spatial position of the input feature map, avoids misplacement of the modulation parameters due to resolution differences, and guarantees the accuracy of the generated disease area; the channel-level disease area identification ensures that the areas of different disease types are independently identified in their respective channels, avoids confusion of multi-disease features in the modulation process, and ensures the purity of single-channel disease features; the spatially differentiated modulation parameter prediction ensures that the disease area and the non-disease area are treated specifically, solves the disease feature diffusion problem caused by traditional normalized global uniform treatment, and improves the authenticity of the generated details; the step-by-step upsampling operation ensures that the feature map resolution is smoothly improved, which can better preserve disease details compared to single large-multiple upsampling, and through the progressive optimization of the M upsampled residual blocks, more rich disease features are accumulated, laying a foundation for generating high-quality target disease images.

[0088] In one specific embodiment, S4 further comprises the following steps:

[0089] K different random noise vectors are obtained, where K is an integer greater than 1.

[0090] Steps S41 to S45 are repeatedly performed for each random noise vector to generate K target disease images.

[0091] The random noise vector is used to generate multiple target disease images with different details but consistent core disease attributes based on different random noise vectors under the same multi-channel condition input tensor graph (i.e., the same disease type, location, and size constraint), thereby maximizing the diversity of noise on the basis of precise condition constraints and solving the problem of single details of a generated image, and providing more abundant sample support for subsequent underground disease detection model training and disease risk identification.

[0092] In an embodiment, the target image generation model includes a target generator and a target discriminator, and the target image generation model is obtained by the following steps:

[0093] S10, obtaining a multi-channel condition input tensor graph corresponding to each real underground disease image according to the real disease attribute information corresponding to each real underground disease image.

[0094] S20, inputting the multi-channel condition input tensor graph corresponding to each real underground disease image and the corresponding random noise vector into a preset generator in a preset image generation model to obtain a reference disease image corresponding to each real underground disease image, wherein the preset image generation model further includes a preset discriminator, and the preset generator is internally provided with a spatial adaptive normalization module.

[0095] S30, performing adversarial training on the preset generator and the preset discriminator according to all real underground disease images and the reference disease image corresponding to each real underground disease image to obtain a trained target generator and a target discriminator.

[0096] The preset image generation model is an initial SPADE-GAN architecture built before training, including a preset generator (with a built-in SPADE module) and a preset discriminator, and the parameters are in a random initialization state, which is used to provide a basic network framework for training. The preset generator is responsible for preliminary image generation, and the preset discriminator is responsible for preliminary distinguishing between true and false, thereby providing a starting point for subsequent adversarial training.

[0097] The method for obtaining the multi-channel condition input tensor graph corresponding to the real underground disease image can refer to the method for obtaining the multi-channel condition input tensor graph corresponding to the target disease attribute information, which will not be described again.

[0098] The reference disease image is an initial generated image output by the preset generator after receiving the multi-channel condition input tensor graph, and the initial real degree is low. With the training iteration, the generated image gradually approaches the real image. As an intermediate product of generator optimization, the generated image is used to calculate the loss difference with the real image, guide the generator parameter update, and gradually improve the generation effect.

[0099] The adversarial training is a game training process of the generator and the discriminator: the generator tries to generate an image that can deceive the discriminator, and the discriminator tries to accurately distinguish the true and false images. The two are alternately optimized through the loss function back propagation, to drive the generator to generate images closer to the real images, and to improve the distinguishing ability of the discriminator, forming a positive cycle of "improving the accuracy of the generator → enhancing the ability of the discriminator → further optimizing the generator".

[0100] The loss function includes an adversarial loss for measuring the distinguishing ability of the discriminator and the deception ability of the generator, and a reconstruction loss for ensuring that the pixel-level structure of the reference image is consistent with the real image. Those skilled in the art know that the adversarial training based on the loss function in the prior art for training the image generation model falls within the protection scope of the present application, and will not be repeated here.

[0101] The above, with the real underground disease image and the corresponding multi-channel condition input tensor graph as the training sample pair, the preset generator learns to generate a disease image conforming to the real characteristics from the condition graph, and the preset discriminator learns to distinguish the real image + condition graph from the generated image + condition graph. Through the adversarial game and loss optimization of the two, the authenticity and condition matching degree of the generated image of the generator are gradually improved, and finally the target generator and the target discriminator capable of accurately generating the target disease image are obtained, so as to integrate the condition information into the adversarial training framework of the image generation model, solve the problem of disease attribute deviation caused by unconstrained generation, and realize the condition controllable and high authenticity image generation.

[0102] The above, by constructing a multi-channel condition input tensor graph, the disease type and spatial attribute are converted into structured condition information, and different disease types and non-zero pixel marks are mapped to different disease areas through different channels, so that the target image generation model can learn and generate corresponding disease characteristics in a targeted manner through different channels, avoid mutual interference and confusion of multiple disease characteristics in the modulation process, and make the generation process realize accurate and controllable disease attribute; through the spatial self-adaptive normalization module built in the generator, the spatial differential modulation parameters are directly predicted by using the multi-channel condition input tensor graph in the feature map generation process, so that the target image generation model can inject disease type and spatial attribute information into the feature map at each intermediate generation stage, and perform targeted processing on the disease area and the non-disease area, which not only preserves the relative feature relationship of the disease area, but also strengthens the high-frequency texture details, avoids the mode collapse and detail loss problem of the traditional image generation model under a small sample, and makes the generated target disease image have high real local features, thereby realizing controllable high-quality image generation according to the user-specified attributes based on a small amount of real samples.

[0103] Embodiment two

[0104] Embodiment two of the present application provides a non-transitory computer readable storage medium, which can be arranged in an electronic device to save at least one instruction or at least one program related to a method in the method embodiment, and the at least one instruction or the at least one program is loaded and executed by the processor to realize the generative model based image generation method provided in the above embodiment.

[0105] Embodiment three

[0106] Embodiment three of the present application provides an electronic device, which includes a processor and the non-transitory computer readable storage medium in embodiment two of the present application.

[0107] The above is only the preferred embodiment of the present application, and does not limit the present application in any form. Although the present application has been disclosed as above with the preferred embodiment, it is not intended to limit the present application. Any person skilled in the art can make some changes or modifications to the above disclosed technical content without departing from the technical solution of the present application, and any simple modification, equivalent change and modification of the above embodiment according to the technical essence of the present application still belong to the scope of the technical solution of the present application.

Claims

1. An image generation method based on a generative model, characterized in that, The image generation method includes: S1, based on the one-to-one correspondence between preset disease types and channel numbers, determine the target channel corresponding to each target disease type in the target disease attribute information, wherein the preset disease types include at least voids, looseness and cracks; S2, based on the spatial attribute information corresponding to each type of target disease in the target disease attribute information, locate the disease area of ​​the target channel corresponding to each type of target disease in the preset multi-channel tensor map, wherein the preset multi-channel tensor map has all zero values, the number of channels of the preset multi-channel tensor map is consistent with the total number of preset disease types, and the spatial resolution of the preset multi-channel tensor map is the same as the spatial resolution of the target disease image to be generated; S3, update the pixel values ​​in all disease areas of the preset multi-channel tensor map to non-zero flag values ​​to obtain a multi-channel conditional input tensor map; S4, the multi-channel conditional input tensor is processed according to the target image generation model to obtain the target disease image. The target generator of the target image generation model has a built-in spatial adaptive normalization module. The spatial adaptive normalization module is used to perform spatial adaptive modulation on the feature map extracted by the target generator according to the disease area in the multi-channel conditional input tensor during the image generation process.

2. The image generation method based on a generative model according to claim 1, characterized in that, The spatial attribute information includes spatial location information, size information, and boundary range information. S2 includes the following steps: S21. Based on the total number of preset disease types and the spatial resolution of the target disease image to be generated, initialize a preset multi-channel tensor image with all zero values. S22, For any target disease type, calculate the coordinates (x1, y1, x2, y2) of the disease area in the corresponding target channel based on the spatial attribute information of the current target disease type, where the disease area is a rectangular area, (x1, y1) is the coordinate of the upper left corner of the disease area, and (x2, y2) is the coordinate of the lower right corner of the disease area; S23, traverse all target disease types and locate the disease area of ​​the target channel corresponding to each target disease type.

3. The image generation method based on a generative model according to claim 1, characterized in that, S3 includes the following steps: S31, update the pixel values ​​in all disease areas of the preset multi-channel tensor image to non-zero flag values, wherein the non-zero flag values ​​are preset fixed values; S32, keep the pixel values ​​in the non-disease area of ​​the preset multi-channel tensor map as zero, and obtain the multi-channel conditional input tensor map.

4. The image generation method based on a generative model according to claim 1, characterized in that, S4 includes the following steps: S41, a random noise vector is input to the fully connected layer in the target generator through the noise input port, and the random noise vector is linearly mapped by the fully connected layer to obtain a one-dimensional initial feature vector; S42, the spatial resolution of the one-dimensional initial feature vector is adjusted by Reshape to obtain an initial feature map, wherein the spatial resolution of the initial feature map is smaller than the spatial resolution of the multi-channel conditional input tensor map; S43, preprocess the multi-channel conditional input tensor graph, and input the preprocessed multi-channel conditional input tensor graph into the M upsampled residual blocks connected in series in the target generator through the conditional input port, where M is an integer greater than 2. The preprocessing includes resolution adjustment, tensor transformation and normalization operations. S44, the initial feature map is input to the first upsampled residual block, and after being modulated and upsampled by M upsampled residual blocks in sequence, a reference feature map output by the Mth upsampled residual block is obtained, wherein the spatial resolution of the reference feature map is the same as the spatial resolution of the multi-channel conditional input tensor map; S45, the reference feature map is convolved and mapped according to the convolutional layer and activation function in the target generator to obtain the target lesion image.

5. The image generation method based on a generative model according to claim 4, characterized in that, Each upsampled residual block contains a spatial adaptive normalization module, and S44 includes the following steps: S441, in the i-th upsampled residual block, the multi-channel conditional input tensor is read according to the adaptive normalization module, and the multi-channel conditional input tensor is downsampled by interpolation to obtain the i-th reference tensor corresponding to the i-th upsampled residual block. The spatial resolution of the i-th reference tensor is the same as the spatial resolution of the input feature map corresponding to the i-th upsampled residual block, i=1,2,...,M. When i=1, the input feature map corresponding to the 1st upsampled residual block is the initial feature map. When i≥2, the input feature map corresponding to the i-th upsampled residual block is the reference feature map corresponding to the (i-1)-th upsampled residual block. S442, Identify the disease area in each channel of the i-th reference tensor map according to the adaptive normalization module; S443, based on the spatial distribution of the diseased area read, predict the modulation parameters of each spatial location in the input feature map corresponding to the i-th upsampled residual block, wherein the modulation parameters include scaling factor and offset factor; S444, the input feature map corresponding to the i-th upsampled residual block is normalized and spatially adaptively modulated using the modulation parameters to obtain the modulated feature map corresponding to the i-th upsampled residual block; S445, the modulated feature map corresponding to the i-th upsampled residual block is upsampled by interpolation to obtain the reference feature map corresponding to the i-th upsampled residual block; S446, traverse all upsampled residual blocks in the order of i=1, 2, ..., M, and obtain the reference feature map output by the Mth upsampled residual block.

6. The image generation method based on a generative model according to claim 4, characterized in that, S4 also includes the following steps: Obtain K distinct random noise vectors, where K is an integer greater than 1; For each random noise vector, repeat steps S41 to S45 to generate K target lesion images.

7. The image generation method based on a generative model according to claim 1, characterized in that, S1 also includes the following steps: S111, acquire several real underground disease images in multiple scenarios, and real disease attribute information corresponding to each real underground disease image, wherein the real disease attribute information includes several real disease types and real spatial attribute information corresponding to the real underground disease images. S112, deduplicate all real disease types in all real disease attribute information to obtain N preset disease types after deduplication, where N≥3; S113 assigns an independent and continuous channel number to each preset disease type, wherein each channel number uniquely corresponds to a channel in a preset multi-channel tensor graph.

8. The image generation method based on a generative model according to claim 7, characterized in that, The target image generation model includes a target generator and a target discriminator, and the target image generation model is obtained through the following steps: S10, Based on the real disease attribute information corresponding to each real underground disease image, obtain the multi-channel conditional input tensor image corresponding to each real underground disease image. S20, input the multi-channel conditional input tensor and the corresponding random noise vector corresponding to each real underground disease image into the preset generator in the preset image generation model to obtain the reference disease image corresponding to each real underground disease image. The preset image generation model also includes a preset discriminator, and the preset generator has a built-in spatial adaptive normalization module. S30: Based on all real underground disease images and the reference disease image corresponding to each real underground disease image, perform adversarial training on the preset generator and preset discriminator to obtain the trained target generator and target discriminator.

9. A non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores at least one instruction or at least one program segment, characterized in that, The at least one instruction or the at least one program segment is loaded and executed by the processor to implement the image generation method based on a generative model as described in any one of claims 1-8.

10. An electronic device, characterized in that, Includes a processor and the non-transitory computer-readable storage medium as described in claim 9.

Images