Picture generation method based on multi-scale features

Abstract

The invention relates to the technical field of picture generation, in particular to a picture generation method based on multi-scale features. The method comprises the following steps: firstly, collecting medical images, patient data and lesion stage information under different equipment and acquisition parameters, after screening preprocessing, constructing and training a multi-scale VQ-VAE model, introducing an attention mechanism, and adopting adaptive codebook updating and multi-codebook fusion quantification; then, a hierarchical autoregression model based on a Transform decoder is constructed and trained, and lesion stage information is fused into the hierarchical autoregression model; and finally, inputting specific lesion stage information, generating a multi-scale discrete index sequence through a hierarchical autoregression model, converting the multi-scale discrete index sequence into a codeword vector through a multi-scale VQ-VAE model, and finally generating a simulated medical image. According to the scheme, the multi-scale VQ-VAE model is utilized to encode the input image into the multi-scale discrete feature representation, and meanwhile, the autoregression model is applied to the discrete hidden space of the VQ-VAE, so that the distribution of the discrete representation sequence can be effectively modeled, and the medical image with higher quality and more realistic sense can be generated.

Description

Technical Field

[0001] The present invention relates to the technical field of image generation, and in particular to a method for generating an image based on multi-scale features. Background Art

[0002] Medical imaging plays a vital role in disease diagnosis, treatment monitoring, and medical research. However, obtaining sufficient and diverse medical imaging data often faces challenges, such as data privacy, high acquisition costs, and sparse data on rare cases. Therefore, using generative models to synthesize high-quality medical images has important research and application value.

[0003] In recent years, significant progress has been made in deep learning-based generative models, such as variational autoencoders (VAEs) and generative adversarial networks (GANs). VQ-VAE, a variant of VAE, introduces a discrete latent space representation, enabling it to learn more structured features and demonstrating potential in image generation. However, traditional VQ-VAEs typically only learn feature representations at a single scale, making it difficult to capture the complex multi-scale structural information found in medical imaging data. Summary of the Invention

[0004] In order to solve the above problems, the present invention provides an image generation method based on multi-scale features, which uses a multi-scale VQ-VAE model to encode the input image into discrete feature representations of different resolutions. At the same time, an autoregressive model is applied to the discrete latent space of VQ-VAE, which can effectively model the distribution of discrete representation sequences, overcome the shortcomings of the existing technology, and generate higher quality and more realistic medical images.

[0005] A method for generating an image based on multi-scale features, comprising:

[0006] S1. Data collection and preprocessing: Collect medical images using different equipment and different acquisition parameters, as well as data corresponding to patients of different races and age groups. At the same time, collect various types of information related to the stage of the disease. Screen the collected data to determine whether it meets the requirements, and then preprocess the data that meets the requirements.

[0007] Preferably, the screening and judging of the collected data to determine whether it meets the requirements specifically includes:

[0008] Image quality assessment: For medical images, the system evaluates whether the resolution reaches the preset minimum resolution threshold, whether the image noise level is within an acceptable range, and whether there are obvious artifacts or missing areas. If the image resolution is lower than the threshold, the noise level is too high, or there are serious artifacts or missing areas, the image data is judged to be non-compliant.

[0009] Data integrity check: Check whether each piece of imaging data corresponds to complete medical parameters, discrete labels, and multimodal information. If key information is missing, the data is deemed to be non-compliant.

[0010] Data consistency verification: Verify the logical consistency between medical parameters. If there is an obvious logical contradiction between the parameters, the data is judged to be non-compliant.

[0011] S2. Build and train a multi-scale VQ-VAE model: Build a multi-scale VQ-VAE model with a hierarchical structure, including an encoder, a vector quantization layer, and a decoder. Use the training data to train the model so that it has the ability to encode, quantize, and decode multi-scale features of medical images.

[0012] Preferably, the multi-scale VQ-VAE model specifically includes:

[0013] Hierarchical encoder: Contains multiple encoder modules, each of which is composed of a stack of convolutional layers, batch normalization layers, and ReLU activation functions, which gradually encodes the input medical image into multiple feature maps of different resolutions;

[0014] Vector quantization layer: An independent vector quantization layer is connected to the output of each encoder module. Each quantization layer contains a learnable codebook and uses L2 distance to quantize the continuous feature vectors of the corresponding resolution into a discrete codeword index sequence;

[0015] Hierarchical decoder: Contains a decoder module that is symmetrical with the encoder structure. It uses deconvolution layers, batch normalization layers, and ReLU activation functions to gradually decode and fuse the received multi-scale discrete feature representations, and finally reconstructs a medical image of the same size as the input.

[0016] Preferably, the encoder module of the hierarchical encoder introduces channel attention and spatial attention mechanisms to improve the efficiency and accuracy of medical image feature extraction, specifically including:

[0017] Channel Attention Mechanism:

[0018] Global information aggregation: For the input feature map, the channel-dimensional feature descriptors are obtained through global average pooling and global maximum pooling operations respectively;

[0019] Feature fusion and nonlinear transformation: The results of global average pooling and global maximum pooling are concatenated in the channel dimension, and then a nonlinear transformation is performed through a fully connected layer;

[0020] Channel weight generation and feature adjustment: Apply the Sigmoid activation function to the features after nonlinear transformation to obtain the channel attention weight, and multiply the weight by the original feature map element-wise in the channel dimension to obtain the feature map adjusted by the channel attention mechanism;

[0021] Spatial Attention Mechanism:

[0022] Feature fusion and convolution operation: The feature map after channel attention adjustment is subjected to average pooling and maximum pooling operations in the channel dimension, and then the two results are spliced in the channel dimension. The spliced result is convolved through a convolution layer to obtain a spatial attention map;

[0023] Spatial weight generation and feature adjustment: Apply the Sigmoid activation function to the above spatial attention map to obtain the spatial attention weight, and multiply the weight by the feature map after channel attention adjustment element by element to obtain the feature map adjusted by the spatial attention mechanism.

[0024] Preferably, the vector quantization layer adopts adaptive codebook updating to adjust the codebook size according to the feature distribution of different lesion stages, specifically including:

[0025] Data grouping and feature acquisition: We collected a dataset of medical images at different disease stages and divided them into early-stage, mid-stage, and late-stage lesions based on the clear classification of the lesion stages. We then used the encoder in the multi-scale VQ-VAE model to perform feature extraction on the medical images within each lesion stage group, obtaining a set of feature vectors at the corresponding scale for each lesion stage.

[0026] Statistical feature calculation: Based on each lesion stage and the corresponding feature vector set at each scale, the standard deviation σ, skewness S, and kurtosis H of each dimension of the feature vector are calculated;

[0027] Complexity evaluation index calculation: Define the feature complexity evaluation index C to measure the complexity of features at different lesion stages. This index is calculated using the formula C = α·σ + β·|S| + γ·H, where α, β, and γ are preset weight coefficients, and α+β+γ=1.

[0028] Determine the size of the initial codebook: Based on the feature complexity evaluation index, determine the number of codewords in the initial codebook, and keep the codeword dimension consistent with the dimension of the feature vector;

[0029] Dynamic optimization of the training process: During the training of the multi-scale VQ-VAE model, the actual effect of feature vector quantization at each lesion stage is continuously monitored. If it is found that a certain lesion stage has obvious feature loss and large feature restoration deviation after quantization, the currently set codebook size does not match the characteristics of this stage. At this time, the codebook size is dynamically adjusted according to the actual situation.

[0030] The formula for calculating the number of codewords in the initial codebook is:

[0031]

[0032] Where K is the number of codewords in the initial codebook, K base is the number of basic codewords set, and δ is a scaling parameter used to control the rate at which the codebook size changes with feature complexity.

[0033] Preferably, the vector quantization layer further uses multi-codebook fusion to quantize the same feature map, specifically including:

[0034] Region division and codebook adaptation: The feature map to be quantized is divided into multiple sub-regions using a grid partitioning method. The feature complexity of the feature vector in each sub-region is calculated. Based on the feature complexity range adapted by each codebook, sub-regions with relatively simple features and small statistical feature variations are assigned to codebooks with relatively few codewords and suitable for low-complexity features. Sub-regions with complex features and large statistical feature fluctuations are assigned to codebooks with a large number of codewords and adapted for high-complexity features.

[0035] Regional quantization: For each subregion's feature vector, the distance between it and each codeword in the corresponding codebook is calculated using the Euclidean distance metric. The codeword with the closest distance is determined as the quantization result of the feature vector, thereby obtaining the quantization index sequence of each subregion in the corresponding codebook.

[0036] Fusion of multi-codebook quantization results: The weight of the sub-region is determined based on the degree to which the sub-region contains the key features of the lesion and the feature complexity of the sub-region. The index sequence obtained by quantizing the sub-region for each codebook is fused using a weighted fusion method to generate a quantization index sequence used to represent the entire feature map.

[0037] Preferably, the method for determining the weight of the sub-region specifically includes:

[0038] Lesion relevance weight calculation: For each sub-region, its relevance score with the lesion is calculated, specifically by calculating the similarity between the sub-region feature vector and the known lesion feature vector template;

[0039] Feature complexity weight calculation: Calculate the feature complexity weight of the sub-region based on its feature complexity evaluation index. The complexity evaluation index is calculated in the same way as when determining the codebook size.

[0040] Comprehensive weight calculation: The comprehensive weight of each sub-region is calculated by combining the lesion correlation weight and feature complexity weight;

[0041] The calculation formula of the comprehensive weight is:

[0042]

[0043] Where W r is the lesion correlation weight, f is the sub-region feature vector, f lesion is the known lesion feature vector template, W c is the feature complexity weight, C is the sub-region feature complexity evaluation index, is the sum of the feature complexity evaluation indicators of all sub-regions in the feature map, W is the comprehensive weight, and w1 and w2 are the preset weight coefficients.

[0044] S3. Build and train a hierarchical autoregressive model: Build a hierarchical autoregressive model based on the Transformer decoder structure, integrate lesion stage-related information into the model, and train it using training data containing medical images of different lesion stages and corresponding encoding information, so that it can generate corresponding multi-scale discrete index sequences based on the lesion stage information.

[0045] S4. Generate medical images: Input specific lesion stage information, generate a multi-scale discrete index sequence through the trained hierarchical autoregressive model, convert it into a codeword vector with the help of the multi-scale VQ-VAE model, and finally use the multi-scale VQ-VAE decoder to generate a medical image that simulates the characteristics of the specific lesion stage.

[0046] Preferably, generating medical images specifically includes:

[0047] Input and encoding: Input specific lesion stage information and encode it into an embedding vector;

[0048] Index sequence generation: Using the trained hierarchical autoregressive model and the lesion stage embedding vector as a condition, a set of multi-scale discrete index sequences is generated by sampling layer by layer from coarse scale to fine scale;

[0049] Codeword vector search: The generated discrete index sequence of each scale is searched for the corresponding codeword vector through the corresponding codebook in the trained multi-scale VQ-VAE model;

[0050] Image generation: The obtained multi-scale codeword vector sequence is input into the trained multi-scale VQ-VAE decoder. The decoder fuses this information to generate a medical image that simulates the characteristics of the specific lesion stage.

[0051] Compared with the prior art, the advantages of the present invention are:

[0052] Multi-scale feature extraction: By building a hierarchical multi-scale VQ-VAE model, it can effectively extract the multi-scale features of medical images, thereby more comprehensively capturing the information of medical images and improving the accuracy and authenticity of generated images;

[0053] Enhanced attention mechanism: Introducing channel attention and spatial attention mechanisms into the hierarchical encoder can adaptively adjust the weights of feature maps, highlight important feature information, and further improve the efficiency and accuracy of feature extraction;

[0054] Adaptive codebook update: The vector quantization layer uses an adaptive codebook update method to adjust the codebook size according to the feature distribution of different lesion stages. This can better adapt to the feature complexity of different lesion stages and reduce feature loss and restoration deviation.

[0055] Multi-codebook fusion quantization: Quantize the same feature map using multi-codebook fusion, which can reasonably allocate codebooks based on the feature complexity and lesion correlation of sub-regions, improving quantization accuracy and flexibility;

[0056] Integrating lesion stage information: The hierarchical autoregressive model incorporates lesion stage-related information into the model, which can generate corresponding medical images based on specific lesion stage information, meeting the needs of medical research and clinical diagnosis for imaging data at different lesion stages. DETAILED DESCRIPTION

[0057] In one embodiment, a specific application process of a multi-scale feature-based image generation method for generating simulated CT images of brain tumors at different lesion stages is provided, including:

[0058] Data Collection:

[0059] Brain CT scan images were collected from imaging databases of multiple hospitals, encompassing images acquired using different CT equipment models and scanning parameters to ensure data diversity. Relevant information about the corresponding patients was also collected, including age, gender, tumor-related medical parameters, discrete labels for lesion stages (classified as early, mid-stage, and late-stage), and available genetic testing data for some patients.

[0060] After screening, a total of 1,000 valid CT image data containing brain tumors were collected to form the original data set.

[0061] Data screening judgment:

[0062] Image quality assessment: Professional image analysis software was used to perform a quality check on each CT image. The minimum resolution threshold was set at 0.5 mm × 0.5 mm per pixel. This review found that 50 images failed to meet the requirements due to low resolution. Image noise levels were measured by calculating the standard deviation of grayscale values. A noise standard deviation threshold of 15 (grayscale values range from 0 to 255) was set. Images exceeding this threshold were considered to have excessive noise, and a total of 30 such images were screened out. Furthermore, the images were manually inspected for obvious artifacts or missing critical brain regions, and an additional 20 images with such issues were excluded.

[0063] Data integrity check: We checked the patient information corresponding to each remaining image one by one and found that 100 images were missing tumor volume data or lesion stage labels, and these data were also determined to be non-compliant;

[0064] Data consistency verification: For images with tumor volume and disease duration data, reasonable association rules were set based on medical experience. For example, early-stage tumors grow relatively slowly. If the tumor volume in an image is too large and the disease duration is too short, which does not conform to normal medical logic, the image data is determined to have consistency issues and 50 such data sets are excluded.

[0065] After the above screening and judgment, 750 brain CT image data that met the requirements and the corresponding complete patient information were finally obtained as the data set for subsequent processing.

[0066] Build and train a multi-scale VQ-VAE model:

[0067] Hierarchical encoder construction:

[0068] A hierarchical encoder consisting of two encoder modules was constructed. The first encoder module used a 3x3 convolution kernel with a stride of 2 for downsampling. The input image was 512×512×1, and after convolution, the output was a feature map of 256×256×64. In this module, the convolution layer was followed by a batch normalization layer and a ReLU activation function. Channel attention and spatial attention mechanisms were introduced. In the channel attention mechanism, the reduction factor r=16 was set, and channel features were captured through global average pooling and maximum pooling operations. Channel attention weights were obtained through processing with a fully connected layer and an activation function, and the original feature map was recalibrated. The spatial attention mechanism used a 7x7 convolution kernel to process the feature map adjusted by channel attention, focusing on key areas such as tumors, generating spatial attention weights, and adjusting the feature map again.

[0069] The second encoder module also continues downsampling with a 3x3 convolution kernel and a stride of 2, further converting the 256×256×64 feature map output by the previous layer into a 128×128×128 feature map. It is also equipped with a batch normalization layer, ReLU activation function, and channel and spatial attention mechanism to extract more abstract features.

[0070] Vector quantization layer configuration:

[0071] The codebook size is adjusted according to the feature distribution of different lesion stages. First, the images in the dataset are grouped according to the lesion stage (early, middle, and late). For each group of images, the constructed encoder is used to extract the feature vector set at each scale. Taking the early lesion stage as an example, the statistical indicators such as the standard deviation σ, skewness S, and kurtosis H of each dimension of the feature vector are calculated, and the feature complexity evaluation index C = α·σ+β·|S|+γ·H is defined (here the weight coefficients α = 0.3, β = 0.4, and γ = 0.4 are set based on experience). The number of basic codewords K base =128, scaling factor δ = 0.5, by formula The codebook size corresponding to the early stage of lesions at the first scale is determined to be 160×64 (the codeword dimension D is set to 64 based on the feature vector dimension). For the intermediate and late stages of lesions, the corresponding codebook sizes are calculated using the same method, and are 256×64 and 384×64, respectively. At the second scale, the codebook sizes adapted to different lesion stages are also calculated based on the feature complexity, such as 128×128 for the early stage, 200×128 for the intermediate stage, and 320×128 for the late stage.

[0072] The same feature map is quantized using multi-codebook fusion. The feature map output by the encoder is divided into 16×16 sub-regions of equal size. The statistical features are calculated for the feature vectors in each sub-region. The sub-regions with relatively simple features are assigned to the codebook adapted for the early lesion stage, and the complex ones are assigned to the codebook adapted for the late lesion stage. Each codebook quantizes the assigned sub-regions respectively. The Euclidean distance is used to measure the distance between the feature vector and the codeword to determine the quantization index sequence. Then, the weight of each sub-region is calculated. For the lesion correlation weight W r , obtained by calculating the cosine similarity between the sub-region feature vector and the known brain tumor feature vector template; the feature complexity weight W c Calculated according to the previous feature complexity evaluation index and corresponding formula; comprehensive weight W = 0.6·W r +0.4W c (Here, w1=0.6, w2=0.3) Finally, the quantization index of each sub-region is weighted and summed according to the weight to obtain a fused quantization index sequence.

[0073] Hierarchical decoder construction:

[0074] The hierarchical decoder is symmetrical to the encoder and consists of two decoder modules. The first decoder module uses a 3x3 convolution kernel and a deconvolution layer with a stride of 2 for upsampling. The feature map corresponding to the input quantized feature vector is restored from 128×128×128 to 256×256×64. During this process, the upsampled feature map is fused with the feature map of the corresponding scale in the encoder through a jump connection method of feature weighted fusion. The fusion weight is dynamically adjusted according to the importance of the feature. After fusion, it is processed by a batch normalization layer and a ReLU activation function.

[0075] The second decoder module continues to upsample with a 3x3 convolution kernel and a deconvolution layer with a stride of 2, restoring the feature map to the original image size of 512×512×1. It also performs feature weighted fusion, batch normalization, and ReLU activation operations to finally reconstruct a brain CT image of the same size as the input. At the same time, a generative adversarial network is introduced in the decoder to construct a discriminator to determine whether the generated image is realistic, forming adversarial training with the decoder to further improve the quality of the generated image.

[0076] Model training:

[0077] Define the loss function, including reconstruction loss, perceptual loss, and structural similarity index loss, and set the initial weights to 0.5, 0.3, and 0.2 respectively.

[0078] The multi-scale VQ-VAE model was trained using the stochastic gradient descent optimization algorithm with a learning rate of 0.001, a momentum of 0.9, and a batch size of 16. During the training process, the validation set (10% of the data was divided from the dataset as the validation set) was evaluated every 10 epochs. Based on the change in the loss value on the validation set, if the validation set loss did not decrease for three consecutive times, the learning rate was appropriately reduced (multiplied by 0.1). A total of 100 epochs were trained until the model converged, and the trained multi-scale VQ-VAE model parameters were saved.

[0079] Build and train a hierarchical autoregressive model:

[0080] Model architecture construction:

[0081] A hierarchical autoregressive model was constructed using a stacked architecture of three Transformer decoders. The first Transformer decoder, conditioned on the lesion stage embedding vector, predicts a discrete index sequence at the coarsest scale (128×128). This decoder incorporates a multi-head self-attention mechanism (set to 8 heads), which performs self-attention on the input lesion stage embedding vector and positional encoding information to capture the associations between different elements. This is followed by a feedforward neural network, which performs nonlinear transformations on the self-attention output to enhance the model's expressiveness.

[0082] The second Transformer decoder receives the output of the first decoder and the lesion stage embedding vector, and predicts a discrete index sequence at a finer scale (256×256). It also has a multi-head self-attention mechanism and a feedforward neural network. A memory mechanism is introduced during the calculation process, using a gated recurrent unit to help the model remember previously predicted index information and better handle long sequence dependencies. At the same time, a multi-scale attention mechanism is used to enable the model to focus on feature information at different scales, fusing and transferring information from coarse to fine scales.

[0083] The third Transformer decoder generates a discrete index sequence at the finest scale (512×512) based on the output of the first two decoders and the lesion stage embedding vector. Its structure is similar to that of the first two decoders, and it further refines the index sequence generation to accurately reflect the detailed characteristics of brain tumors at different lesion stages.

[0084] Lesion stage information coding:

[0085] The information representing the stage of the disease, such as tumor volume, disease duration, discrete labels of the disease stage, and the patient's genetic data, is encoded into an embedding vector of fixed dimension (set to 256 dimensions) through a learnable embedding layer.

[0086] A dynamic embedding layer is used to dynamically adjust the generation method of the embedding vector according to the input lesion stage information.

[0087] Conditional input mechanism implementation:

[0088] By splicing, the lesion stage embedding vector is concatenated with the input embedding of the Transformer decoder in dimension, so that the lesion stage information is directly integrated into the decoder input;

[0089] At the same time, the attention mechanism is used to use the lesion stage embedding vector as context information to guide the generation of the index sequence. In the calculation process of the multi-head self-attention mechanism of each Transformer decoder, the lesion stage embedding vector is used as additional query information. By calculating the attention weight, the model can dynamically pay attention to the relevant features according to the lesion stage information when generating each position of the index sequence, and focus more on generating index values that match the current lesion stage.

[0090] Model training:

[0091] The training data used 750 real brain CT images at different disease stages and their multi-scale discrete index sequences encoded by the trained multi-scale VQ-VAE model, as well as corresponding disease stage labels, tumor-related parameters, and genetic data.

[0092] A multi-task loss function is defined, including negative log-likelihood loss, perceptual loss, and structural similarity index loss, with initial weights set to 0.4, 0.3, and 0.3, respectively.

[0093] During the training process, the Adam optimization algorithm is used, the learning rate is set to 0.0001, β1 = 0.9, β2 = 0.999, the batch size is 8, and the validation set (10% of the training data is divided as the validation set) is evaluated every 5 epochs. The weights of different loss functions are adaptively adjusted according to the change of the loss value on the validation set. For example, if the negative log-likelihood loss decreases slowly in several consecutive validation set evaluations, while the perceptual loss or structural similarity index loss decreases significantly, the weights of the latter two are appropriately increased, and the weight of the negative log-likelihood loss is reduced. A total of 80 epochs are trained until the model converges, and the trained hierarchical autoregressive model parameters are saved.

[0094] Generate medical images simulating specific stages of a lesion:

[0095] Enter specific lesion stage information and code:

[0096] Suppose we want to generate a simulated mid-stage brain tumor CT image. We input the corresponding lesion stage information, including the tumor volume range, disease duration, discrete labels of the mid-stage lesions, and some simulated genetic data.

[0097] Through the previously trained lesion stage information encoding method, these input information are encoded into a 256-dimensional embedding vector, which contains the comprehensive feature information of the mid-stage lesion, serving as the conditional information for subsequent image generation.

[0098] Index sequence generation:

[0099] Using the trained hierarchical autoregressive model, starting from the coarsest scale (128×128), the first Transformer decoder generates a corresponding discrete index sequence based on the embedding vector of the intermediate lesion stage and its internal computation mechanism. This sequence reflects the approximate distribution of brain tumor characteristics at this coarse scale.

[0100] Next, the second Transformer decoder generates a discrete index sequence at a finer scale (256×256) based on the output of the previous decoder and the lesion stage embedding vector, further refining the tumor features. For example, the preliminary morphology of the tumor edge begins to be reflected.

[0101] Finally, the third Transformer decoder generates a discrete index sequence at the finest scale (512×512) based on the outputs of the first two decoders and the lesion stage embedding vector. This index sequence is now able to more fully characterize the size, location, internal structure, and other characteristic information of brain tumors in the mid-stage lesion.

[0102] Codeword vector search and image generation:

[0103] The generated discrete index sequence of each scale is used to find the corresponding codeword vector through the corresponding codebook in the previously trained multi-scale VQ-VAE model;

[0104] The obtained multi-scale codeword vector sequence is input into the decoder of the multi-scale VQ-VAE model. The decoder gradually fuses and processes the codeword vector information according to the set fusion method, activation function (ReLU) and upsampling operation, and finally generates a CT scan image of a brain tumor that simulates the characteristics of the mid-term lesion stage. The image shows the size, position, density and internal structure of the brain tumor in the mid-term lesion stage, which can be used for doctor training, diagnostic reference and other purposes.

[0105] Throughout this specification, references to terms such as "one embodiment," "example," or "specific example" indicate that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0106] The preferred embodiments of the present invention disclosed above are intended only to help illustrate the present invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the present invention to the specific embodiments described. Obviously, many modifications and variations are possible based on the content of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention, thereby enabling those skilled in the art to better understand and utilize the present invention. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A method for generating an image based on multi-scale features, characterized in that: include: S1. Data Collection and Preprocessing: Collect medical images acquired with different equipment and parameters, as well as data from patients of different races and age groups. Also collect information related to the stage of the disease. Screen the collected data to determine if it meets the requirements, and then preprocess the data that meets the requirements. S2. Build and train a multi-scale VQ-VAE model: Build a multi-scale VQ-VAE model with a hierarchical structure, including an encoder, a vector quantization layer, and a decoder. Train the model using training data to enable it to encode, quantize, and decode multi-scale features of medical images. S3. Build and train a hierarchical autoregressive model: Build a hierarchical autoregressive model based on the Transformer decoder structure, incorporate information related to the lesion stage into the model, and train it using training data containing medical images of different lesion stages and corresponding encoding information, so that it can generate corresponding multi-scale discrete index sequences based on the lesion stage information; S4. Generate medical images: Input specific lesion stage information, generate a multi-scale discrete index sequence through the trained hierarchical autoregressive model, convert it into a codeword vector with the help of the multi-scale VQ-VAE model, and finally use the multi-scale VQ-VAE decoder to generate a medical image that simulates the characteristics of the specific lesion stage.

2. The method for generating an image based on multi-scale features according to claim 1, wherein: The screening and judgment of the collected data to determine whether it meets the requirements specifically includes: Image quality assessment: For medical images, the system evaluates whether the resolution reaches the preset minimum resolution threshold, whether the image noise level is within an acceptable range, and whether there are obvious artifacts or missing areas. If the image resolution is lower than the threshold, the noise level is too high, or there are serious artifacts or missing areas, the image data is judged to be non-compliant. Data integrity check: Check whether each piece of imaging data corresponds to complete medical parameters, discrete labels, and multimodal information. If key information is missing, the data is deemed to be non-compliant. Data consistency verification: Verify the logical consistency between medical parameters. If there is an obvious logical contradiction between the parameters, the data is judged to be non-compliant.

3. The method for generating an image based on multi-scale features according to claim 1, wherein: The multi-scale VQ-VAE model specifically includes: Hierarchical encoder: Contains multiple encoder modules, each of which is composed of a stack of convolutional layers, batch normalization layers, and ReLU activation functions, which gradually encodes the input medical image into multiple feature maps of different resolutions; Vector quantization layer: An independent vector quantization layer is connected to the output of each encoder module. Each quantization layer contains a learnable codebook and uses L2 distance to quantize the continuous feature vectors of the corresponding resolution into a discrete codeword index sequence; Hierarchical decoder: Contains a decoder module that is symmetrical with the encoder structure. It uses deconvolution layers, batch normalization layers, and ReLU activation functions to gradually decode and fuse the received multi-scale discrete feature representations, and finally reconstructs a medical image of the same size as the input.

4. The method for generating an image based on multi-scale features according to claim 3, wherein: The encoder module of the hierarchical encoder introduces channel attention and spatial attention mechanisms to improve the efficiency and accuracy of medical image feature extraction, specifically including: Channel Attention Mechanism: Global information aggregation: For the input feature map, the channel-dimensional feature descriptors are obtained through global average pooling and global maximum pooling operations respectively; Feature fusion and nonlinear transformation: The results of global average pooling and global maximum pooling are concatenated in the channel dimension, and then a nonlinear transformation is performed through a fully connected layer; Channel weight generation and feature adjustment: Apply the Sigmoid activation function to the features after nonlinear transformation to obtain the channel attention weight, and multiply the weight by the original feature map element-wise in the channel dimension to obtain the feature map adjusted by the channel attention mechanism; Spatial Attention Mechanism: Feature fusion and convolution operation: The feature map after channel attention adjustment is subjected to average pooling and maximum pooling operations in the channel dimension, and then the two results are spliced in the channel dimension. The spliced result is convolved through a convolution layer to obtain a spatial attention map; Spatial weight generation and feature adjustment: Apply the Sigmoid activation function to the above spatial attention map to obtain the spatial attention weight, and multiply the weight by the feature map after channel attention adjustment element by element to obtain the feature map adjusted by the spatial attention mechanism.

5. The method for generating an image based on multi-scale features according to claim 3, wherein: The vector quantization layer uses adaptive codebook updating to adjust the codebook size according to the feature distribution of different lesion stages, specifically including: Data grouping and feature acquisition: We collected a dataset of medical images at different disease stages and divided them into early-stage, mid-stage, and late-stage lesions based on the clear classification of the lesion stages. We then used the encoder in the multi-scale VQ-VAE model to perform feature extraction on the medical images within each lesion stage group, obtaining a set of feature vectors at the corresponding scale for each lesion stage. Statistical feature calculation: Based on each lesion stage and the corresponding feature vector set at each scale, the standard deviation σ, skewness S, and kurtosis H of each dimension of the feature vector are calculated; Complexity evaluation index calculation: Define the feature complexity evaluation index C to measure the complexity of features at different lesion stages. This index is calculated using the formula C = α·σ + β·|S| + γ·H, where α, β, and γ are preset weight coefficients, and α+β+γ=1. Determine the size of the initial codebook: Based on the feature complexity evaluation index, determine the number of codewords in the initial codebook, and keep the codeword dimension consistent with the dimension of the feature vector; Dynamic optimization of the training process: During the training of the multi-scale VQ-VAE model, the actual effect of feature vector quantization at each lesion stage is continuously monitored. If it is found that a certain lesion stage has obvious feature loss and large feature restoration deviation after quantization, the currently set codebook size does not match the characteristics of this stage. At this time, the codebook size is dynamically adjusted according to the actual situation. The formula for calculating the number of codewords in the initial codebook is: Where K is the number of codewords in the initial codebook, K base is the number of basic codewords set, and δ is a scaling parameter used to control the rate at which the codebook size changes with feature complexity.

6. The method for generating an image based on multi-scale features according to claim 3, wherein: The vector quantization layer also uses multi-codebook fusion to quantize the same feature map, specifically including: Region division and codebook adaptation: The feature map to be quantized is divided into multiple sub-regions using a grid partitioning method. The feature complexity of the feature vector in each sub-region is calculated. Based on the feature complexity range adapted by each codebook, sub-regions with relatively simple features and small statistical feature variations are assigned to codebooks with relatively few codewords and suitable for low-complexity features. Sub-regions with complex features and large statistical feature fluctuations are assigned to codebooks with a large number of codewords and adapted for high-complexity features. Regional quantization: For each subregion's feature vector, the distance between it and each codeword in the corresponding codebook is calculated using the Euclidean distance metric. The codeword with the closest distance is determined as the quantization result of the feature vector, thereby obtaining the quantization index sequence of each subregion in the corresponding codebook. Fusion of multi-codebook quantization results: The weight of the sub-region is determined based on the degree to which the sub-region contains the key features of the lesion and the feature complexity of the sub-region. The index sequence obtained by quantizing the sub-region for each codebook is fused using a weighted fusion method to generate a quantization index sequence used to represent the entire feature map.

7. The method for generating an image based on multi-scale features according to claim 6, wherein: The method for determining the weight of the sub-region specifically includes: Lesion relevance weight calculation: For each sub-region, its relevance score with the lesion is calculated, specifically by calculating the similarity between the sub-region feature vector and the known lesion feature vector template; Feature complexity weight calculation: Calculate the feature complexity weight of the sub-region based on its feature complexity evaluation index. The complexity evaluation index is calculated in the same way as when determining the codebook size. Comprehensive weight calculation: The comprehensive weight of each sub-region is calculated by combining the lesion correlation weight and feature complexity weight; The calculation formula of the comprehensive weight is: Where W r is the lesion correlation weight, f is the sub-region feature vector, f lesion is the known lesion feature vector template, W c is the feature complexity weight, C is the sub-region feature complexity evaluation index, is the sum of the feature complexity evaluation indicators of all sub-regions in the feature map, W is the comprehensive weight, and w1 and w2 are the preset weight coefficients.

8. The method for generating an image based on multi-scale features according to claim 1, wherein: Generating medical images specifically includes: Input and encoding: Input specific lesion stage information and encode it into an embedding vector; Index sequence generation: Using the trained hierarchical autoregressive model and the lesion stage embedding vector as a condition, a set of multi-scale discrete index sequences is generated by sampling layer by layer from coarse scale to fine scale; Codeword vector search: The generated discrete index sequence of each scale is searched for the corresponding codeword vector through the corresponding codebook in the trained multi-scale VQ-VAE model; Image generation: The obtained multi-scale codeword vector sequence is input into the trained multi-scale VQ-VAE decoder. The decoder fuses this information to generate a medical image that simulates the characteristics of the specific lesion stage.

Images