Extreme environment degradation image simulation generation method and device

Abstract

Embodiments of the present disclosure disclose an extreme environment degradation image simulation generation method and device. A specific implementation of the method comprises: inputting a degraded image into a diffusion autoencoder to obtain a latent image feature vector; generating a foreground prior visual feature vector set and a background prior visual feature vector; inputting the foreground prior visual feature vector set and the background prior visual feature vector into a double-end prototype resampling model to obtain a foreground visual perception label feature vector set and a background visual perception label feature vector; generating a context decoupling feature vector set; inputting the context decoupling feature vector set into a context topology relationship coordination model and then into a dynamic semantic relationship aggregation model to obtain a context semantic feature vector; and performing decoding processing on the context semantic feature vector and the context decoupling feature vector set after weighted fusion to obtain a target degraded image. The implementation can improve the quality of images in extreme degradation scenarios.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of computer technology, specifically to a method and apparatus for simulating and generating images of extreme environmental degradation. Background Technology

[0002] Currently, images acquired in extreme environments (e.g., low light, remote sensing, underwater, foggy and rainy weather) are of low quality and have limited data volume, resulting in poor performance as auxiliary training data sources for downstream visual models. Visual data resources are scarce, making the generation of images in extreme degradation scenarios an increasingly important issue. The common approach for simulating the generation of extreme environment degradation images is to use an L2I (Layout-to-Image) diffusion model to encode the geometric layout and label information corresponding to the degradation image into position-aware and category-aware markers. These markers are then input into a latent diffusion space. Finally, multi-instance masking is performed on the geometric layout information to generate the extreme environment degradation image.

[0003] However, in practice, it has been found that when using the above method to simulate and generate degraded images in extreme environments, the following technical problem often exists: In existing complex extreme environment degradation scenarios, due to the high visual coupling between foreground instances and backgrounds due to their similar appearance, and the frequent spatial occlusion between foreground instances, the L2I diffusion model does not take into account the occlusion problems between foreground instances and between foreground instances and backgrounds, resulting in poor quality of the generated degraded images.

[0004] The information disclosed in this background section is only intended to enhance the understanding of the background of the present disclosure concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0006] Some embodiments of this disclosure propose a method and apparatus for simulating and generating images of extreme environmental degradation, in order to solve one or more of the technical problems mentioned in the background section above.

[0007] In a first aspect, some embodiments of this disclosure provide a method for simulating and generating degraded images in extreme environments, comprising: acquiring a degraded image, text prompt information, and object geometric layout information under extreme conditions; inputting the degraded image into a diffusion variational autoencoder to obtain a latent image feature vector; generating a foreground prior visual feature vector set and a background prior visual feature vector based on the text prompt information and the object geometric layout information; inputting the foreground prior visual feature vector set and the background prior visual feature vector into a dual-end prototype resampling model to obtain a foreground visual perception marker feature vector set and a background visual perception marker feature vector; and generating a foreground prior visual feature vector set and a background visual perception marker feature vector based on the latent image feature vector. The aforementioned object geometric layout information, the aforementioned foreground visual perception marker feature vector set, the aforementioned text prompt information, and the aforementioned background visual perception marker feature vector are used to generate a context decoupling feature vector set. This context decoupling feature vector set is then input into a context topology coordination model to obtain a context topology correction feature vector set. This context topology correction feature vector set is then input into a dynamic semantic relationship aggregation model to obtain a context semantic feature vector. The context semantic feature vector and the aforementioned context decoupling feature vector set are then weighted and fused to obtain a degraded visual feature vector. Finally, the degraded visual feature vector is decoded to obtain the target degraded image.

[0008] Secondly, some embodiments of this disclosure provide an extreme environment degradation image simulation generation apparatus, comprising: an acquisition unit configured to acquire a degradation image, text prompt information, and object geometric layout information under extreme conditions; a first input unit configured to input the degradation image into a diffusion variational autoencoder to obtain a latent image feature vector; a first generation unit configured to generate a foreground visual prior feature vector set and a background visual prior feature vector based on the text prompt information and the object geometric layout information; a second input unit configured to input the foreground visual prior feature vector set and the background visual prior feature vector into a dual-end prototype resampling model to obtain a foreground visual perception marker feature vector set and a background visual perception marker feature vector; and a second generation unit configured to generate a foreground visual prior feature vector set and a background visual perception marker feature vector based on the text prompt information and the object geometric layout information. The image feature vector, the aforementioned object geometric layout information, the aforementioned foreground visual perception marker feature vector set, the aforementioned text prompt information, and the aforementioned background visual perception marker feature vector are used to generate a context decoupling feature vector set; the third input unit is configured to input the aforementioned context decoupling feature vector set into a context topology coordination model to obtain a context topology correction feature vector set; the fourth input unit is configured to input the aforementioned context topology correction feature vector set into a dynamic semantic relationship aggregation model to obtain a context semantic feature vector; the fusion unit is configured to perform weighted fusion processing on the aforementioned context semantic feature vector and the aforementioned context decoupling feature vector set to obtain a degraded visual feature vector; and the decoding unit is configured to decode the aforementioned degraded visual feature vector to obtain a target degraded image.

[0009] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, such that when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0010] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method as described in any implementation of the first aspect.

[0011] The above embodiments of this disclosure have the following beneficial effects: the extreme environment degradation image simulation generation method of some embodiments of this disclosure can improve the image quality under extreme degradation scenarios. Specifically, the reason for the poor quality of the generated degradation images is that, under existing complex extreme environment degradation scenarios, there is a high degree of visual coupling between foreground instances and background due to appearance similarity, and frequent spatial occlusion between foreground instances. The L2I diffusion model does not consider the occlusion problems between foreground instances and between foreground instances and background, resulting in poor quality of the generated degradation images. Based on this, the extreme environment degradation image simulation generation method of some embodiments of this disclosure can first obtain the degradation image, text prompt information, and object geometric layout information under extreme environment. Here, the degradation image provides real degradation scene visual information, the text prompt information describes the global semantic features of the scene, and the object geometric layout information accurately locates the position and category of foreground instances through bounding boxes, providing spatial constraints and semantic guidance for the subsequent generation process. Secondly, the above degradation image is input into a diffusion variational autoencoder to obtain a latent image feature vector. Here, the diffusion variational autoencoder compresses the degraded image into a low-dimensional latent feature vector to obtain basic global visual information of the degraded image, providing foundational information for the subsequent generation of context-decoupled feature vectors. Next, based on the aforementioned text prompts and object geometric layout information, a foreground prior visual feature vector set and a background prior visual feature vector are generated. Here, visual prior information of the same category as the foreground instance is obtained through object geometric layout information, and prior information of the background is obtained through text prompts, supplementing detailed information in extreme environments and solving the blurring problem in extreme degradation scenarios. Then, the aforementioned foreground prior visual feature vector set and background prior visual feature vector are input into a dual-end prototype resampling model to obtain a foreground visual perception labeled feature vector set and a background visual perception labeled feature vector. Here, the dual-end prototype resampling model includes a foreground instance prototype resampling model and a background resampling model to extract rich visual structural information of the foreground and background from the foreground instance and background, thereby enhancing the perception of visual differences between the foreground and background. Then, based on the aforementioned latent image feature vectors, object geometric layout information, foreground visual perception marker feature vector set, text prompt information, and background visual perception marker feature vectors, a context decoupling feature vector set is generated. Here, the context decoupling feature vector set includes foreground decoupling feature vector sets and background decoupling feature vectors. The interaction between foreground instances and background and latent image feature vectors is decoupled through parallel decoupling of foreground instances and background. Subsequently, the aforementioned context decoupling feature vector set is input into the context topology coordination model to obtain the context topology correction feature vector set.Here, the topological relationship coordination model can construct a topological graph including at least one foreground instance and background, and then extract and strengthen the occlusion connections of the context to suppress irrelevant and redundant connections, thereby correcting the mutual topological relationships of the context. Then, the aforementioned context topological relationship correction feature vector set is input into the dynamic semantic relationship aggregation model to obtain the context semantic feature vector. Here, the dynamic semantic relationship aggregation model can aggregate the commonalities and differences in semantic information of contexts corresponding to different weights under different categories, thereby improving the comprehensiveness and visual detail of the context semantic information. Then, the aforementioned context semantic feature vector and the aforementioned context decoupling feature vector set are weighted and fused to obtain the degraded visual feature vector. Here, by assigning different attention weights to the foreground instance and background, more visual detail information can be extracted, thereby increasing the amount of detailed visual information included in the degraded visual feature vector, so that subsequent decoding can generate a higher-quality degraded image. Finally, the aforementioned degraded visual feature vector is decoded to obtain the target degraded image. Therefore, this extreme environment degraded image simulation generation method can improve the image quality in extreme degraded scenarios. Attached Figure Description

[0012] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0013] Figure 1 This is a flowchart of some embodiments of the extreme environment degradation image simulation generation method according to the present disclosure;

[0014] Figure 2 This is a schematic diagram illustrating the transformation of topological occlusion relationships between foreground instances in some embodiments of the extreme environment degradation image simulation generation method according to this disclosure;

[0015] Figure 3 These are comparative schematic diagrams of generated environmental degradation images in some embodiments of the extreme environment degradation image simulation generation method according to this disclosure;

[0016] Figure 4 These are schematic diagrams illustrating the structure of some embodiments of the extreme environment degradation image simulation generation apparatus according to this disclosure;

[0017] Figure 5 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0018] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0019] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0020] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0021] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0022] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0023] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0024] Figure 1 A flow 100 of some embodiments of a degraded image generation method according to the present disclosure is shown. The degraded image generation method includes the following steps:

[0025] Step 101: Obtain degraded images, text prompts, and object geometric layout information under extreme conditions.

[0026] In some embodiments, the execution entity (e.g., an electronic device) of the above-described extreme environment degradation image simulation generation method can acquire degradation images, text prompts, and object geometric layout information via wired or wireless connections. The degradation image can be an image created under extreme conditions where detailed or structural information of individual objects in the image is lost or damaged. For example, extreme degradation scenarios can be, but are not limited to, at least one of the following: underwater environment, foggy or rainy weather, remote sensing imagery, artifacts, low light, or blur. The text prompts can be natural language statements describing the overall visual appearance of the degradation scene. The object geometric layout information can be the positional and semantic information of each object in the degradation image. For example, the degradation image can be an image of a turtle and a jellyfish in an underwater low-light environment; the positional information can be the coordinates of the upper left and lower right corners of the bounding boxes of the turtle and jellyfish, and the semantic information can be the semantic category information of the turtle and jellyfish.

[0027] Step 102: Input the degraded image into the diffusion variational autoencoder to obtain the latent image feature vector.

[0028] In some embodiments, the execution entity may input the degraded image into a diffusion variational autoencoder to obtain a latent image feature vector. This latent image feature vector can characterize information such as the texture, color, and geometric shape distribution of the degraded image. For example, the degraded image may be a turtle and jellyfish in a low-light underwater environment, and the latent image feature vector may include turtle shell outline information and jellyfish tentacles transparency information. Similarly, the degraded image may be a vehicle in hazy weather, and the latent image feature vector may include vehicle outline information and fog concentration gradient information.

[0029] Step 103: Generate a foreground prior visual feature vector set and a background prior visual feature vector set based on the text prompt information and the object's geometric layout information.

[0030] In some embodiments, the executing entity can generate a foreground prior visual feature vector set and a background prior visual feature vector based on the text prompt information and the object geometric layout information. The foreground prior visual feature vector in the foreground prior visual feature vector set can represent the geometric visual information of existing objects in the degraded image that belong to the same category as a foreground object in the degraded image, as found in the foreground object candidate dictionary. The foreground object candidate dictionary can be a database used to store visual feature information of various foreground objects. The background prior visual feature vector can represent the overall scene information of the environment and background information surrounding each foreground instance in the degraded image. For example, a foreground object can be a turtle, and the foreground prior visual feature vector can include, but is not limited to, at least one of the following: turtle edge information, texture information, lighting information, and geometric layout position information.

[0031] As an example, the aforementioned execution entity can first retrieve at least one image of the same category from the RUOD dataset (Rethinking general Underwater Object Detection) based on the semantic information of the object's geometric layout. Secondly, it extracts the foreground information feature vectors of these images using ResNet-50 (Residual Network 50) as a foreground prior visual feature vector set. Finally, it inputs the aforementioned text prompt information into the SDXL (Stable Diffusion XL) model to obtain the background image. Then, it extracts the background prior visual feature vectors of the background image using DINOv2 (DIstillation with NO labels v2).

[0032] In some optional implementations of certain embodiments, generating the foreground prior visual feature vector set and the background prior visual feature vector based on the aforementioned text prompt information and the aforementioned object geometric layout information may include the following steps:

[0033] The first step involves retrieving candidate degradation feature instance groups from a pre-defined candidate instance dictionary that spatially and semantically match the aforementioned object's geometric layout information. These are then used as target candidate instance groups, resulting in a target candidate instance set. The pre-defined candidate instance dictionary can be a pre-constructed offline database storing images of foreground objects according to semantic categories. The target candidate instances in the target candidate instance set can be instance images from the pre-defined candidate instance dictionary that match the aforementioned semantic information. For example, if the semantic information is a turtle, the target candidate instance set can include, but is not limited to, at least one of the following: low-light turtle images, turtle shell images, and underwater turtle belly images.

[0034] The second step involves performing the following average pooling steps based on the aforementioned target candidate instance set:

[0035] Sub-step 1 involves performing image processing on each target candidate instance in the aforementioned target candidate instance set to generate processed candidate instances, thus obtaining a processed candidate instance set. The processed candidate instances in this set can be images generated by stitching together the processed results of instance images matching the semantic information after undergoing different image processing operations. For example, the instance image matching the semantic information can be a tortoise shell image. The different image processing operations can include, but are not limited to, at least one of the following: image processing to extract the binary image of the tortoise shell contour using the Canny operator; image processing to extract the tortoise shell convexity texture image using Gabor filtering; image processing to extract the brightness distribution map of the dark area using grayscale transformation; and image stitching of the tortoise shell images. The image stitching of the tortoise shell image can be a process of stitching together the tortoise shell image, the binary image of the tortoise shell contour, the tortoise shell convexity texture image, and the brightness distribution map of the dark area.

[0036] Sub-step 2 involves inputting the processed candidate instance set into an image convolutional feature extraction model to obtain a foreground instance visual feature vector set. The foreground instance visual feature vectors in this set represent the visual information of the foreground instances. The image convolutional feature extraction model can be a model that takes the processed candidate instance set as input, selectively extracts significant visual features, and outputs the foreground instance visual feature vector set. For example, the image convolutional feature extraction model can be, but is not limited to, at least one of the following: a deformable convolutional neural network, or a dynamic region-aware convolutional model. For example, if the processed candidate instance is a turtle image, the foreground instance visual feature vectors in the foreground instance visual feature vector set can include, but are not limited to, at least one of the following: turtle shell texture features, and turtle shell edge contour information.

[0037] Sub-step 3 involves performing average pooling on the aforementioned set of visual feature vectors for foreground instances to obtain a set of prior visual feature vectors for the foreground.

[0038] The third step involves retrieving training background images from a pre-set training background image set that match the text prompts corresponding to the aforementioned degraded images, and using these images as target training background images. This pre-set training background image set can be a pre-trained set of background images under various extreme environments. In practice, the executing entity can utilize the CLIP (Contrastive Language-Image Pre-training, a multimodal pre-trained model) model to match and filter images from the pre-set training background image set based on the text prompt information to obtain the target training background image.

[0039] The fourth step is to determine the above target training background image as the target candidate instance set, and then perform the above average pooling step again, and use the obtained foreground prior visual feature vector set as the background prior visual feature vector.

[0040] Step 104: Input the foreground prior visual feature vector set and the background prior visual feature vector into the dual-end prototype resampling model to obtain the foreground visual perception label feature vector set and the background visual perception label feature vector.

[0041] In some embodiments, the aforementioned execution entity can input the foreground prior visual feature vector set and the background prior visual feature vector into a two-end prototype resampling model to obtain a foreground visual perception labeled feature vector set and a background visual perception labeled feature vector. The two-end prototype resampling model can perform feature projection, resampling processing, and dynamic adaptation operations on the input foreground prior visual feature vector set and background prior visual feature vector to output a model containing both the foreground visual perception labeled feature vector set and the background visual perception labeled feature vector. For example, the two-end prototype resampling model can be a Transformer decoder model. The foreground visual perception labeled feature vector in the foreground visual perception labeled feature vector set can represent the visual information of foreground instances in the degraded image and information obtained through interactive training with the two-end prototype resampling model. The background visual perception labeled feature vector can represent the visual information of the background in the environmental image and information obtained through interactive training with the two-end prototype resampling model. For example, the aforementioned degraded image could be an image of a turtle and jellyfish in an underwater, low-light coral reef environment. In this scenario, the foreground visual perception marker feature vector could include, but is not limited to, at least one of the following: texture information of the turtle's dorsal markings under low light, color distortion correction information caused by water scattering, and light refraction pattern information of the jellyfish's tentacles' transparency underwater. The aforementioned background visual perception marker feature vector could include, but is not limited to, at least one of the following: blurry outline distribution information of the coral reef community, and scattering path information of the light source in the water.

[0042] In some optional implementations of certain embodiments, the aforementioned dual-end prototype resampling model includes a foreground prototype resampling model and a background prototype resampling model. The foreground prototype resampling model takes as input the foreground prior visual feature vector set and the foreground learnable query label, and outputs a foreground visual perception label feature vector set. The background prototype resampling model takes as input the background prior visual feature vector and the background learnable query label, and outputs a context visual perception label feature vector. The foreground prototype resampling model may include four network layers, each layer comprising a cross-attention mechanism and a feedforward neural network stacked in series. The background prototype resampling model may have the same network structure as the foreground prototype resampling model, but with different inputs, outputs, and model parameters.

[0043] Optionally, inputting the foreground prior visual feature vector set and the background prior visual feature vector into the dual-end prototype resampling model to obtain the foreground visual perception label feature vector set and the background visual perception label feature vector may include the following steps:

[0044] The first step is to obtain the foreground learnable query labels corresponding to the aforementioned foreground prior visual feature vector set, and the background learnable query labels corresponding to the aforementioned background prior visual feature vector set. The foreground learnable query labels can be randomly initialized and trainable parameter matrices corresponding to the aforementioned foreground prototype resampling model. The background learnable query label information can be randomly initialized and trainable parameter matrices corresponding to the aforementioned background prototype resampling model.

[0045] The second step involves performing a linear transformation on the aforementioned foreground visual feature vector set to obtain a foreground prior key vector set and a foreground prior value vector set. The foreground prior key vectors in the foreground prior key vector set can be represented as retrievable identifier feature vectors containing abstract semantic feature information of the foreground category and feature information of key visual attributes. The foreground prior value vectors in the foreground prior value vector set can be represented as containing richer and more specific visual feature information of the foreground category. For example, in an image of a puppy on grass, where the foreground category is "puppy," the foreground prior key vectors may include, but are not limited to, at least one of the following: semantic label information representing "dog," visual information representing "standing posture," and the foreground prior value vectors may include, but are not limited to, at least one of the following: morphological local feature information of erect ears, and fur color texture information of black hair.

[0046] The third step involves inputting the aforementioned foreground learnable query tags, the aforementioned foreground prior key vector set, and the aforementioned foreground prior value vector set into the aforementioned foreground prototype resampling model to obtain the foreground visual perception tag feature vector set.

[0047] The fourth step involves performing a linear transformation on the aforementioned background visual feature vectors to obtain background prior key vectors and background prior value vectors. The background prior key vectors in the aforementioned background prior key vector set can be represented as retrievable identifier feature vectors containing abstract semantic feature information of the background category and feature information of key visual attributes. The background prior value vectors in the aforementioned background prior value vector set can be represented as visual feature information containing richer and more specific background information. For example, in an image of a dog on grass, where the background is grass, the aforementioned background prior key vectors may include, but are not limited to, at least one of the following: scene semantic information representing "grass," and structural feature visual information representing "horizontal extension." The aforementioned background prior value vectors may include, but are not limited to, at least one of the following: visual information representing short, dense green grass and scattered wildflowers, and spatial regularity information representing clear textures in the foreground and blurred textures in the background.

[0048] Fifth, input the above-mentioned background learnable query tags, the above-mentioned background prior key vectors, and the above-mentioned background prior value vectors into the above-mentioned background prototype resampling model to obtain the background visual perception tag feature vector.

[0049] Step 105: Generate a context decoupling feature vector set based on the potential image feature vector, object geometric layout information, foreground visual perception marker feature vector set, text prompt information, and background visual perception marker feature vector.

[0050] In some embodiments, the executing entity can generate a context decoupling feature vector set based on the potential image feature vector, the object geometric layout information, the foreground visual perception marker feature vector set, the text prompt information, and the background visual perception marker feature vector. The context decoupling feature vectors in the context decoupling feature vector set can characterize the relationship between foreground instances and the background.

[0051] In addressing the first technical problem mentioned above, a second technical problem often arises: considering only the decoupling between foreground instances and the background leads to unclear extraction of basic visual information from the degraded image when identifying foreground instances and the background, resulting in low-quality generated degraded images. A conventional solution to this second technical problem is to interact with latent image features through single-path decoupling attention on foreground instances to extract richer basic visual information from the degraded image. However, this conventional solution still suffers from the following problem: degraded information in the background region is not extracted independently, resulting in unclear background visual information and thus low-quality generated degraded images. Considering the shortcomings of the conventional solution and leveraging the advantages / current technology of feature interaction decoupling between foreground feature vectors, background feature vectors, and latent feature vectors possessed by our company, we have decided to adopt the following solution:

[0052] In some optional implementations of certain embodiments, the aforementioned object geometric layout information includes: a foreground instance bounding box position information set and a foreground instance semantic category information set. The foreground instance bounding box position information in the foreground instance bounding box position information set may be the coordinates of the top-left and bottom-right corners of the foreground instance bounding box. The foreground instance semantic category information in the foreground instance semantic category information set may be the category name information of the foreground instance.

[0053] Optionally, the process of generating a context-decoupled feature vector set based on the latent image feature vector, the object geometric layout information, the foreground visual perception marker feature vector set, the text prompt information, and the background visual perception marker feature vector, and then generating a target degraded image based on the context-decoupled feature vector set, may include the following steps:

[0054] The first step is to perform text encoding on the aforementioned foreground instance semantic category information set to obtain a foreground category semantic feature vector set. The foreground category semantic feature vectors in this set can be represented as semantic vector representations of the category information to which the object name belongs. For example, if the semantic category is "car," the foreground category semantic feature vectors in the set could be vectors representing the basic semantic information of a car, such as windows, tires, and a metal shell. In practice, the executing entity can use the CLIP model to perform text encoding on the aforementioned foreground instance semantic category information set to obtain the foreground category semantic feature vector set.

[0055] The second step involves encoding the bounding box positions of the foreground instance bounding box location information set to obtain a set of bounding box position feature vectors. These bounding box position feature vectors can be represented as numerical codes of the object's position coordinates. In practice, the executing entity can utilize Fourier embedding to transform the foreground instance bounding box location information set, obtaining Fourier-transformed bounding box position embedding vectors. Then, it can use an MLP (Multilayer Perceptron) model for feature extraction to obtain the bounding box position feature vector set.

[0056] The third step involves concatenating the foreground visual perception marker feature vector set, the foreground category semantic feature vector set, and the bounding box position feature vector set along the feature dimension to obtain the foreground concatenated feature vector set. The foreground concatenated feature vectors in this set can be characterized as a comprehensive vector representation combining volume visual features, category semantics, and positional information.

[0057] The fourth step involves performing a linear transformation on the aforementioned latent image feature vectors to obtain the image visual transformation feature vectors. These image visual transformation feature vectors can be query vectors generated from the aforementioned latent image feature vectors through a linear projection layer, used as feature vectors for retrieving key information within the attention mechanism.

[0058] Fifth, the aforementioned image visual transformation feature vectors and the aforementioned foreground stitching feature vector set are input into the multi-head cross-attention mechanism layer to obtain the foreground visual attention feature vector set. The foreground visual attention feature vectors in this set can be represented as visual semantic information of the foreground instance that integrates global environmental features.

[0059] Step 6: Determine the bounding box mask vector set of the aforementioned foreground instance bounding box location information set. The bounding box mask vectors in this set can represent the location identifiers of the spatial regions of the foreground object in binary form. For example, the foreground instance could be a car bounding box with coordinates [120, 80, 300, 200]. The bounding box mask vectors in this set could be 1 for bounding box coordinates x∈[120, 300], y∈[80, 200], and 0 for other regions. In practice, the executing entity can utilize an attention masking mechanism to determine the bounding box mask vector set of the aforementioned foreground instance bounding box location information set.

[0060] Step 7: Determine the element-wise product of each foreground visual attention feature vector in the foreground visual attention feature vector set and the corresponding bounding box mask vector in the bounding box mask vector set to obtain the foreground instance layout interaction feature vector set.

[0061] Step 8 involves text encoding the aforementioned text prompt information to obtain a text prompt feature vector. This text prompt feature vector can represent semantic information describing the global scene. For example, the text prompt information could be "City streets in dense fog, visibility less than 100 meters," outputting a vector representing dense fog, high fog density, and low visibility. In practice, the executing entity can utilize the CLIP model to perform text-to-numerical vector conversion on the text prompt information to obtain the aforementioned text prompt feature vector.

[0062] Step nine involves concatenating the background visual perception marker feature vector and the text prompt feature vector along the feature dimension to obtain the background concatenated feature vector. This background concatenated feature vector set can be represented as a fusion of degraded visual information and scene semantic information.

[0063] Step 10: Input the aforementioned image visual transformation feature vector and the aforementioned background stitching feature vector into the multi-head cross-attention mechanism layer to obtain the background visual attention feature vector. The aforementioned background visual attention feature vector can be represented as the semantic information resulting from the attention-weighted fusion of text prompt information and image semantic information.

[0064] Step 11: Generate a background region mask vector based on the bounding box mask vector set described above. This background region mask vector represents the location information of the background region in the degraded image, excluding the foreground instance bounding boxes.

[0065] As an example, the aforementioned execution entity can utilize the foreground instance separation decoupling function to generate a background region mask vector based on the aforementioned bounding box mask vector set. The foreground instance separation decoupling function can be:

[0066]

[0067] Where N represents the total number of bounding boxes for the foreground instances mentioned above. M i This represents the binarized attention mask of the i-th bounding box in the foreground instance above.

[0068] Step 12: Determine the element-wise product of the aforementioned background visual attention feature vector and background region mask vector to obtain the background text interaction feature vector. This background text interaction feature vector can be characterized as background descriptive information that matches the degraded environmental features of the background region as described in the text. For example, the aforementioned text prompt information could be set in hazy weather, and the aforementioned background descriptive information could include, but is not limited to, at least one of the following: gradual changes in road fog concentration, gray-white tone features of the sky, and regional restriction information.

[0069] Step 13: Combine the foreground instance context layout interaction feature vector set and the background text interaction feature vector set to obtain the context decoupling feature vector set. In practice, the executing entity can store the foreground instance context layout interaction feature vector set and the background text interaction feature vector set into a preset set to obtain the context decoupling feature vector set. The preset set can be a pre-designed set for storing feature vectors.

[0070] Step fourteen: Generate the target degraded image based on the decoupled feature vector set according to the context. As an example, the implementation of this step can be referred to in sections 106-109, and will not be repeated here.

[0071] Steps one through fourteen above, and their related content, constitute an inventive point of this disclosure, solving the second technical problem mentioned in the background art: "Considering only the decoupling between foreground instances and the background leads to unclear extraction of basic visual information of the degraded image when identifying foreground instances and the background, resulting in low quality of the generated degraded image." The factors leading to unclear extraction of basic visual information of the degraded image when identifying foreground instances and the background, resulting in low quality of the generated environmental degraded image, are often as follows: Only performing feature interaction on a single path for the foreground instance, without independently processing the basic visual information of the background, results in unclear background visual information of the degraded image, leading to low quality of the generated degraded image. Solving these factors allows for the extraction of more basic visual information of the image, improving the realism and detail restoration of the degraded image, and obtaining a high-quality degraded image. To achieve this effect, this disclosure first uses the CLIP model to encode the semantic category information of the foreground instance into a foreground category semantic feature vector containing semantic information, providing guidance for object detail extraction in the foreground instance decoupling path. Fourier embedding is used to transform the bounding box position information of the foreground instance, and a bounding box position feature vector is generated by MLP to lock the range of the foreground instance. By fusing foreground visual perception marker feature vectors, foreground category semantic feature vectors, and bounding box position feature vectors, a foreground stitching feature vector set is formed as the decoupling basis information, preventing environmental information from being mixed in. Next, information is extracted independently through the foreground instance path. The interaction between the latent image feature vector (after feature projection) and the decoupling basis information set is used to extract foreground instance detail information. A binary bounding box mask is generated based on the foreground instance bounding box position information, and the feature modification range is strictly limited by element-wise multiplication to obtain the foreground instance layout interaction feature vector set, ensuring that foreground instance detail information does not affect the background region. Then, information is extracted independently through the background path. CLIP-encoded text prompt information is used to extract basic environmental visual information, resulting in a text prompt feature vector. The background visual markers and text prompt feature vectors are stitched together to construct the background stitching feature vector. Multi-head cross-attention is used to calculate the interaction between the latent image feature vector (after feature projection) and the background stitching feature vector set to extract basic background visual detail information. A reverse masking operation is used to obtain the background text interaction feature vector, ensuring that background effects only affect the background region and avoid impacting the foreground instance region. Subsequently, the foreground instance layout interaction feature vector set and the background text interaction feature vector set are concatenated to obtain the context decoupling feature vector set. By combining the information extracted independently from the foreground instance path and the information extracted independently from the background path, interaction decoupling with the potential feature vectors is performed from two aspects, which can improve the accuracy of interaction decoupling, remove redundant information from the context decoupling feature vector set, and improve the quality of the context decoupling feature vector set.Finally, the decoder is used to reconstruct the context-decoupled feature vector set, resulting in a degraded image with richer visual information, greater realism, and higher detail reproduction.

[0072] Step 106: Input the context decoupling feature vector set into the context topology relationship coordination model to obtain the context topology relationship correction feature vector set.

[0073] In some embodiments, the aforementioned execution entity inputs the aforementioned context decoupling feature vector set into a context topology relationship coordination model to obtain a context topology relationship correction feature vector set. The aforementioned context topology relationship coordination model can be a model that takes the context decoupling feature vector set as input, performs feature adjacency matrix construction and topological feature reprojection operations, and outputs a context topology relationship correction feature vector set. The aforementioned context topology relationship coordination model includes: a relationship coordination model and a graph convolutional neural network model. The context topology relationship correction feature vectors in the aforementioned context topology relationship correction feature vector set can represent the occlusion association information and hierarchical information between foreground instances and background, and between foreground instances. For example... Figure 2 As shown, this illustrates the information on inter-instance occlusion associations represented by the aforementioned context topology correction feature vector set.

[0074] In some optional implementations of certain embodiments, inputting the aforementioned context decoupling feature vector set into the context topology coordination model to obtain the context topology correction feature vector set may include the following steps:

[0075] The first step is to determine the context adjacency matrix of the aforementioned context decoupling feature vector set using the relationship coordination model described above. This relationship coordination model can be a model that takes the aforementioned context decoupling feature vector set as input, performs similarity calculations between context decoupling feature vectors in the set, and outputs the context adjacency matrix of the aforementioned context decoupling feature vector set. The context adjacency matrix can represent the spatial occlusion and semantic association strength information between foreground instances and between foreground instances and the background. For example, the aforementioned degraded image could be a turtle and a jellyfish in an underwater low-light environment, with the jellyfish occluding the turtle's image. An element in the instance adjacency matrix could be a similarity of 0.7 between the turtle and the jellyfish, indicating a strong spatial association between them. The formula corresponding to the aforementioned relationship coordination model is:

[0076]

[0077] Where A represents the aforementioned context adjacency matrix. H j This represents the j-th context decoupling feature vector. Let represent the transpose of the i-th context-decoupling feature vector. |||| represents the magnitude of the context-decoupling feature vector.

[0078] The second step involves using the graph convolutional neural network model described above to perform graph-symmetric normalization on the context adjacency matrix, thereby obtaining a set of context topology correction feature vectors.

[0079] The formula corresponding to the above graph convolutional neural network model is:

[0080]

[0081] in, This represents the set of feature vectors corrected for contextual topology relationships. σ() represents the ReLU (Rectified LinearUnit) activation function. This represents the adjacency relationship between nodes in a graph that have self-loops. The calculation formula can be:

[0082] Υ represents the identity matrix. express The matrix has a dimension of |N+1|×|N+1|. express The degree matrix, Indicates to Sum each row. express The element in row i and column j.

[0083] Step 107: Input the context topology relationship correction feature vector set into the dynamic semantic relationship aggregation model to obtain the context instance semantic feature vector.

[0084] In some embodiments, the execution entity inputs the aforementioned context topology correction feature vector set into a dynamic semantic relationship aggregation model to obtain a context semantic feature vector. The dynamic semantic relationship aggregation model can be a model that takes the context topology correction feature vector set as input, performs gated routing weighting, expert feature extraction, and semantic aggregation operations, and outputs a context semantic feature vector. The dynamic semantic relationship aggregation model includes: a gated network, multiple expert networks, and an aggregation operation network. The aggregation network can be a network that weights and sums the outputs of at least two expert networks activated by the output of the gated network. The context semantic feature vector can characterize the shared and differentiated semantic attributes of objects belonging to the same category as each foreground instance and background, aggregated according to different aggregation weights.

[0085] In some optional implementations of certain embodiments, inputting the aforementioned context topology correction feature vector set into a dynamic semantic relationship aggregation model to obtain a context semantic feature vector may include the following steps:

[0086] The first step is to obtain a normally distributed noise vector. This normally distributed noise vector can represent noise information that conforms to a normal distribution. For example, the normally distributed noise vector may include, but is not limited to, at least one of the following: a feature vector representing simulated water flow information, or a feature vector representing light spot jitter information.

[0087] The second step involves inputting the aforementioned normally distributed noise vector and the aforementioned context topology correction feature vector set into the aforementioned gating network to obtain the expert activation probability weight set for the aforementioned multiple expert networks. The aforementioned gating network can be a model that takes the aforementioned normally distributed noise vector and the aforementioned context topology correction feature vector set as input, performs noise weighting score calculation and probability mapping operations, and outputs the expert activation probability weight set for the aforementioned multiple expert networks. For example, the aforementioned gating network can be a noisy sparse gating model. The expert activation probability weights of the expert networks in the aforementioned expert activation probability weight set can be the probability weight values ​​of expert selection.

[0088] The third step is to select at least two expert activation probability weights from the aforementioned set of expert activation probability weights that satisfy a preset activation probability condition. This preset activation probability condition can be the condition of selecting the expert activation probability weight with the highest value among the first preset number of expert activation probability weights. This preset number can be a pre-defined value. For example, the preset number could be 2.

[0089] The fourth step is to identify at least two expert networks from the multiple expert networks mentioned above that correspond to the activation probability weights of the at least two experts.

[0090] Fifth, the aforementioned context topology correction feature vector set is input into at least two expert networks to obtain a context semantic aggregation feature vector set. The expert network can be a model that takes the context topology correction feature vector set as input, performs semantic feature enhancement processing, and outputs the context semantic aggregation feature vector set. For example, the expert network can be a multilayer perceptron model with two linear layers and a nonlinear ReLU activation function. The context semantic aggregation feature vectors in the aforementioned context semantic aggregation feature vector set can represent the semantic information or common and differential attributes of foreground instances belonging to the same category extracted by an expert network.

[0091] Step 6: Input the aforementioned contextual semantic aggregated feature vector set and the aforementioned activation probability weights of at least two experts into the aforementioned aggregation operation network to obtain the contextual semantic feature vector. The aforementioned aggregation operation network can be a network that takes the aforementioned contextual aggregated feature vector set and the aforementioned activation probability weights of at least two experts as input, performs a weighted feature fusion operation, and outputs the contextual semantic feature vector.

[0092] Step 108: Perform weighted fusion processing on the context semantic feature vector and the context decoupling feature vector set to obtain the degraded visual feature vector.

[0093] In some embodiments, the execution entity performs a weighted fusion process on the aforementioned contextual semantic feature vector and the aforementioned contextual decoupling feature vector set to obtain a degraded visual feature vector. This degraded visual feature vector can represent occlusion information and more detail information of foreground instances and backgrounds within the same category, providing clearer image quality and more detailed texture information than the latent image feature vector. The degraded visual feature vector can represent, but is not limited to, at least one of the following: object attribute information, environmental parameter information, inter-object occlusion logic, and semantic correlation.

[0094] In some optional implementations of certain embodiments, the weighted fusion of the aforementioned context semantic feature vector and the aforementioned context decoupling feature vector set to obtain the degraded visual feature vector may include the following steps:

[0095] The first step is to normalize the aforementioned contextual semantic feature vectors to obtain the contextual semantic relationship aggregation weights. These aggregation weights can be represented as the semantic importance weights of the context. In practice, the executing entity can use the softmax (normalization exponent) function to normalize the contextual semantic feature vectors to obtain the aggregation weights.

[0096] The second step involves element-wise multiplying the aggregated weights of the aforementioned contextual semantic relationships with each contextual decoupling feature vector in the aforementioned contextual decoupling feature vector set to obtain the contextual weight feature vector set. This contextual weight feature vector set can be represented as contextual feature information weighted by semantic importance.

[0097] The third step is to accumulate the above context weight feature vector set to obtain the degraded visual feature vector.

[0098] Step 109: Decode the degraded visual feature vector to obtain the target degraded image.

[0099] In some embodiments, the execution entity decodes the degraded visual feature vector to obtain a target degraded image. The target degraded image can be a high-fidelity degraded image that is spatially and semantically well aligned with a given layout. In practice, the execution entity can utilize the decoder in a diffusion variational autoencoder to decode the degraded visual feature vector to obtain the target degraded image. The diffusion variational autoencoder model, the dual-end prototype resampling model, the context topology coordination model, and the dynamic semantic relationship aggregation model can be various component network models of the degraded network generation model. The degraded network generation model can be trained using a loss function. The loss function can be:

[0100]

[0101] Among them, L VPEC Let θ' represent the loss function of the degenerate network generative model. θ′ represents the trainable parameters. θ represents the frozen parameters of the pre-trained latent diffusion model, and ξ represents the expected value. z0 represents the latent image feature vector. ε~N(0,1) represents standard Gaussian noise (mean 0, variance 1). t represents the diffusion time step. Γ represents the text prompt information. ε represents randomly generated Gaussian noise. Ω(·) represents the noise prediction network of the degenerate network generative model. t Let B represent the latent image feature vector at time step t with noise added. Let B represent the set of positional information in the object's geometric layout information. Let Q represent the set of foreground visual perception marker feature vectors and the set of background visual perception marker feature vectors. This represents the square of the L2 norm, i.e., the mean square error.

[0102] like Figure 3 The diagram shows a comparison of degraded images generated by the aforementioned degradation network generation model, Layout-Diffusion model, MIGC (Multi-instance Generation Controller) model, and CC-Diff (Enhancing Contextual Coherence in Remote Sensing Image Synthesis) model based on the geometric layout information of each degraded image and the corresponding object in multiple degraded images. VPEC can be a degradation network generation model.

[0103] The above embodiments of this disclosure have the following beneficial effects: the extreme environment degradation image simulation generation method of some embodiments of this disclosure can improve the image quality under extreme degradation scenarios. Specifically, the reason for the poor quality of the generated degradation images is that, under existing complex extreme environment degradation scenarios, there is a high degree of visual coupling between foreground instances and background due to appearance similarity, and frequent spatial occlusion between foreground instances. The L2I diffusion model does not consider the occlusion problems between foreground instances and between foreground instances and background, resulting in poor quality of the generated degradation images. Based on this, the extreme environment degradation image simulation generation method of some embodiments of this disclosure can first obtain the degradation image, text prompt information, and object geometric layout information under extreme environment. Here, the degradation image provides real degradation scene visual information, the text prompt information describes the global semantic features of the scene, and the object geometric layout information accurately locates the position and category of foreground instances through bounding boxes, providing spatial constraints and semantic guidance for the subsequent generation process. Secondly, the above degradation image is input into a diffusion variational autoencoder to obtain a latent image feature vector. Here, the diffusion variational autoencoder compresses the degraded image into a low-dimensional latent feature vector to obtain basic global visual information of the degraded image, providing foundational information for the subsequent generation of context-decoupled feature vectors. Next, based on the aforementioned text prompts and object geometric layout information, a foreground prior visual feature vector set and a background prior visual feature vector are generated. Here, visual prior information of the same category as the foreground instance is obtained through object geometric layout information, and prior information of the background is obtained through text prompts, supplementing detailed information in extreme environments and solving the blurring problem in extreme degradation scenarios. Then, the aforementioned foreground prior visual feature vector set and background prior visual feature vector are input into a dual-end prototype resampling model to obtain a foreground visual perception labeled feature vector set and a background visual perception labeled feature vector. Here, the dual-end prototype resampling model includes a foreground instance prototype resampling model and a background resampling model to extract rich visual structural information of the foreground and background from the foreground instance and background, thereby enhancing the perception of visual differences between the foreground and background. Then, based on the aforementioned latent image feature vectors, object geometric layout information, foreground visual perception marker feature vector set, text prompt information, and background visual perception marker feature vectors, a context decoupling feature vector set is generated. Here, the context decoupling feature vector set includes foreground decoupling feature vector sets and background decoupling feature vectors. The interaction between foreground instances and background and latent image feature vectors is decoupled through parallel decoupling of foreground instances and background. Subsequently, the aforementioned context decoupling feature vector set is input into the context topology coordination model to obtain the context topology correction feature vector set.Here, the topological relationship coordination model can construct a topological graph including at least one foreground instance and background, and then extract and strengthen the occlusion connections of the context to suppress irrelevant and redundant connections, thereby correcting the mutual topological relationships of the context. Then, the aforementioned context topological relationship correction feature vector set is input into the dynamic semantic relationship aggregation model to obtain the context semantic feature vector. Here, the dynamic semantic relationship aggregation model can aggregate the commonalities and differences in semantic information of contexts corresponding to different weights under different categories, thereby improving the comprehensiveness and visual detail of the context semantic information. Then, the aforementioned context semantic feature vector and the aforementioned context decoupling feature vector set are weighted and fused to obtain the degraded visual feature vector. Here, by assigning different attention weights to the foreground instance and background, more visual detail information can be extracted, thereby increasing the amount of detailed visual information included in the degraded visual feature vector, so that subsequent decoding can generate a higher-quality degraded image. Finally, the aforementioned degraded visual feature vector is decoded to obtain the target degraded image. Therefore, this extreme environment degraded image simulation generation method can improve the image quality in extreme degraded scenarios.

[0104] Further reference Figure 4 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of an extreme environment degradation image simulation generation apparatus, which are similar to... Figure 1 Corresponding to the method embodiments shown, this extreme environment degradation image simulation generation device can be specifically applied to various electronic devices.

[0105] like Figure 4As shown, an extreme environment degradation image simulation generation device 400 includes: an acquisition unit 401, a first input unit 402, a first generation unit 403, a second input unit 404, a second generation unit 405, a fourth input unit 407, a fusion unit 408, and a decoding unit 409. The acquisition unit 401 is configured to acquire a degradation image, text prompt information, and object geometric layout information under extreme conditions. The first input unit 402 is configured to input the degradation image into a diffusion variational autoencoder to obtain a latent image feature vector. The first generation unit 403 is configured to generate a foreground prior visual feature vector set and a background prior visual feature vector based on the text prompt information and the object geometric layout information. The second input unit 404 is configured to input the foreground prior visual feature vector set and the background prior visual feature vector into a dual-end prototype resampling model to obtain a foreground visual perception marker feature vector set and a background visual perception marker feature vector. The second generation unit 405 is configured to generate a context decoupling feature vector set based on the aforementioned latent image feature vector, the aforementioned object geometric layout information, the aforementioned foreground visual perception marker feature vector set, the aforementioned text prompt information, and the aforementioned background visual perception marker feature vector. The third input unit 406 is configured to input the aforementioned context decoupling feature vector set into a context topology coordination model to obtain a context topology correction feature vector set. The fourth input unit 407 is configured to input the aforementioned context topology correction feature vector set into a dynamic semantic relationship aggregation model to obtain a context semantic feature vector. The fusion unit 408 is configured to perform a weighted fusion process on the aforementioned context semantic feature vector and the aforementioned context decoupling feature vector set to obtain a degraded visual feature vector. The decoding unit 409 is configured to decode the aforementioned degraded visual feature vector to obtain a target degraded image.

[0106] It is understandable that the units and references described in the extreme environment degradation image simulation generation device 400 are... Figure 1 The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the extreme environment degradation image simulation generation device 400 and the units contained therein, and will not be repeated here.

[0107] The following is for reference. Figure 5 It shows a schematic diagram of the structure of an electronic device (e.g., an electronic device) 500 suitable for implementing some embodiments of the present disclosure. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0108] like Figure 5As shown, the electronic device 500 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 508 into a random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0109] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 5 Each box shown can represent a device or multiple devices as needed.

[0110] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 509, or installed from storage device 508, or installed from ROM 502. When the computer program is executed by processing device 501, it performs the functions defined in the methods of some embodiments of this disclosure.

[0111] It should be noted that, in some embodiments of this disclosure, the computer-readable medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0112] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0113] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: acquire a degraded image, text prompt information, and object geometric layout information under extreme conditions; input the aforementioned degraded image into a diffusion variational autoencoder to obtain a latent image feature vector; generate a foreground prior visual feature vector set and a background prior visual feature vector based on the aforementioned text prompt information and the aforementioned object geometric layout information; input the aforementioned foreground prior visual feature vector set and the aforementioned background prior visual feature vector into a dual-end prototype resampling model to obtain a foreground visual perception marker feature vector set and a background visual perception marker feature vector; and, based on the aforementioned latent... Image feature vectors, the aforementioned object geometric layout information, the aforementioned foreground visual perception marker feature vector set, the aforementioned text prompt information, and the aforementioned background visual perception marker feature vectors are used to generate a context decoupling feature vector set. This context decoupling feature vector set is then input into a context topology coordination model to obtain a context topology correction feature vector set. This context topology correction feature vector set is then input into a dynamic semantic relationship aggregation model to obtain a context semantic feature vector. The context semantic feature vector and the aforementioned context decoupling feature vector set are then weighted and fused to obtain a degraded visual feature vector. Finally, the degraded visual feature vector is decoded to obtain the target degraded image.

[0114] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0115] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0116] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including an acquisition unit, a first input unit, a first generation unit, a second input unit, a second generation unit, a third input unit, a fourth input unit, a fusion unit, and a decoding unit. The names of these units do not necessarily limit the specific unit; for example, the acquisition unit may also be described as "a unit for acquiring degraded images, text prompts, and object geometric layout information under extreme conditions."

[0117] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0118] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A method for simulating and generating images of extreme environmental degradation, comprising: Acquire degraded images, text prompts, and object geometry information under extreme environments; The degraded image is input into a diffusion variational autoencoder to obtain a latent image feature vector; Based on the text prompt information and the object geometric layout information, generate a foreground prior visual feature vector set and a background prior visual feature vector; The foreground prior visual feature vector set and the background prior visual feature vector are input into the dual-end prototype resampling model to obtain the foreground visual perception label feature vector set and the background visual perception label feature vector. A context decoupling feature vector set is generated based on the potential image feature vector, the object geometric layout information, the foreground visual perception marker feature vector set, the text prompt information, and the background visual perception marker feature vector; The context decoupling feature vector set is input into the context topology coordination model to obtain the context topology correction feature vector set; The context topology relationship correction feature vector set is input into the dynamic semantic relationship aggregation model to obtain the context semantic feature vector; The context semantic feature vector and the context decoupling feature vector set are weighted and fused to obtain the degraded visual feature vector. The degraded visual feature vectors are decoded to obtain the target degraded image; The dual-end prototype resampling model includes: a foreground prototype resampling model and a background prototype resampling model; and The step of inputting the foreground prior visual feature vector set and the background prior visual feature vector into a dual-end prototype resampling model to obtain the foreground visual perception marker feature vector set and the background visual perception marker feature vector includes: Obtain the foreground learnable query label corresponding to the foreground prior visual feature vector set, and the background learnable query label corresponding to the background prior visual feature vector; The foreground prior visual feature vector set is linearly transformed to obtain the foreground prior key vector set and the foreground prior value vector set. The foreground learnable query tag, the foreground prior key vector set, and the foreground prior value vector set are input into the foreground prototype resampling model to obtain the foreground visual perception tag feature vector set. The foreground prototype resampling model includes: multiple cross-attention layers and multiple feedforward neural networks. The background prior visual feature vector is linearly transformed to obtain the background prior key vector and the background prior value vector. The background learnable query tag, the background prior key vector, and the background prior value vector are input into the background prototype resampling model to obtain the background visual perception tag feature vector.

2. The method according to claim 1, wherein, The step of generating a foreground prior visual feature vector set and a background prior visual feature vector set based on the text prompt information and the object geometric layout information includes: Retrieve candidate instance groups from the preset candidate instance dictionary that are in the same category as the semantic information included in the geometric layout information of the object, and use them as target candidate instance groups to obtain the target candidate instance group set; Based on the target candidate instance set, perform the following average pooling steps: Image processing is performed on each target candidate instance in the target candidate instance set to generate processed candidate instances, thus obtaining the processed candidate instance set. The processed candidate instance set is input into the image convolutional feature extraction model to obtain the foreground instance visual feature vector set. The foreground instance visual feature vector set is subjected to average pooling to obtain the foreground prior visual feature vector set; Retrieve a training background image from a preset training background image set that matches the text prompt corresponding to the degraded image, and use it as the target training background image; The target training background image is determined as the target candidate instance set to perform the average pooling step again, and the resulting foreground prior visual feature vector set is used as the background prior visual feature vector.

3. The method according to claim 1, wherein, The context topology relationship coordination model includes: a relationship coordination model and a graph convolutional neural network model; and The step of inputting the context decoupling feature vector set into the context topology relationship coordination model to obtain the context topology relationship correction feature vector set includes: The context adjacency relation matrix of the context decoupling feature vector set is determined using the relationship coordination model. The context adjacency matrix is ​​graph-symmetric normalized using the graph convolutional neural network model to obtain a context topology correction feature vector set.

4. The method according to claim 1, wherein, The dynamic semantic relation aggregation model includes: a gated network, multiple expert networks, and an aggregation operation network; and The step of inputting the context topology correction feature vector set into the dynamic semantic relationship aggregation model to obtain the context semantic feature vector includes: Obtain the normally distributed noise vector; The normally distributed noise vector and the context topology correction feature vector set are input into the gated network to obtain the expert activation probability weight set for the multiple expert networks; Select at least two expert activation probability weights that satisfy the preset activation probability conditions from the set of expert activation probability weights; Determine at least two expert networks corresponding to the at least two expert activation probability weights from the plurality of expert networks; The context topology relationship correction feature vector set is input into the at least two expert networks to obtain the context semantic aggregation feature vector set; The context semantic aggregated feature vector set and the activation probability weights of at least two experts are input into the aggregation operation network to obtain the context semantic feature vector.

5. The method according to claim 1, wherein, The weighted fusion process of the context semantic feature vector and the context decoupling feature vector set to obtain the degraded visual feature vector includes: The context semantic feature vector is normalized to obtain the context semantic relationship aggregation weight; The context semantic relationship aggregation weights are multiplied element-wise with each context decoupling feature vector in the context decoupling feature vector set to obtain the context weight feature vector set; The context weight feature vector set is accumulated to obtain the degraded visual feature vector.

6. An apparatus for simulating and generating images of extreme environmental degradation, comprising: The acquisition unit is configured to acquire degraded images, text prompts, and object geometry information under extreme conditions; The first input unit is configured to input the degraded image into a diffusion variational autoencoder to obtain a latent image feature vector. The first generation unit is configured to generate a foreground prior visual feature vector set and a background prior visual feature vector based on the text prompt information and the object geometric layout information. The second input unit is configured to input the foreground prior visual feature vector set and the background prior visual feature vector into the dual-end prototype resampling model to obtain the foreground visual perception marker feature vector set and the background visual perception marker feature vector. The second generation unit is configured to generate a context decoupling feature vector set based on the potential image feature vector, the object geometric layout information, the foreground visual perception marker feature vector set, the text prompt information, and the background visual perception marker feature vector; The third input unit is configured to input the context decoupling feature vector set into the context topology coordination model to obtain the context topology correction feature vector set. The fourth input unit is configured to input the context topology relationship correction feature vector set into the dynamic semantic relationship aggregation model to obtain the context semantic feature vector; The fusion unit is configured to perform a weighted fusion process on the context semantic feature vector and the context decoupling feature vector set to obtain a degraded visual feature vector. The decoding unit is configured to decode the degraded visual feature vector to obtain the target degraded image; The dual-end prototype resampling model includes a foreground prototype resampling model and a background prototype resampling model. The second input unit is further configured to: Obtain the foreground learnable query label corresponding to the foreground prior visual feature vector set, and the background learnable query label corresponding to the background prior visual feature vector; The foreground prior visual feature vector set is linearly transformed to obtain the foreground prior key vector set and the foreground prior value vector set. The foreground learnable query tag, the foreground prior key vector set, and the foreground prior value vector set are input into the foreground prototype resampling model to obtain the foreground visual perception tag feature vector set. The foreground prototype resampling model includes: multiple cross-attention layers and multiple feedforward neural networks. The background prior visual feature vector is linearly transformed to obtain the background prior key vector and the background prior value vector. The background learnable query tag, the background prior key vector, and the background prior value vector are input into the background prototype resampling model to obtain the background visual perception tag feature vector.

7. An electronic device, comprising: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-5.

8. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-5.

Images