Information-controllable remote sensing target image simulation generation method
Through the three-dimensional point cloud scanning and semantic segmentation model combined with high-pass filter, the text-image diffusion model is used to generate high-reality target images with multiple angles and multiple postures in remote sensing images, solving the slow generation speed and low realism in the existing technology. and the problem of uncontrollable information, efficient and automated remote sensing target image simulation is achieved.
Patent Information
- Application Number
- CN202411089763.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-11-15
AI Technical Summary
Existing remote sensing image simulation technology is difficult to ensure the generation speed, image scale, target authenticity and information controllability at the same time, and cannot meet the real-time generation needs of large-scale and diversified information remote sensing target images.
The three-dimensional point cloud scanning technology is used to generate the target three-dimensional model, and combined with the semantic segmentation model and high-pass filter, generate multi-angle, multi-pose high-reality target images through feature extraction and detail extraction, and use the pre-trained text-image diffusion model. In the background, the targets are synthesized to achieve full-process automation and information-controllable simulation generation.
Remote sensing target image simulation with full process automation, independent selection of target locations, high generation efficiency, image authenticity and diversity is realized, diversified feature learning data is provided, and the training data quality of intelligent algorithms is improved.
Smart Images

Figure CN118967860A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of image processing, and more specifically, relates to an information-controllable remote sensing target image simulation generation method. Background Art
[0002] With the increase in the number of remote sensing satellites in orbit and their imaging capabilities, the amount of multi-source remote sensing image data generated every day is huge. The pressure of data processing and downlink has increased dramatically, resulting in the inability to use remote sensing image information efficiently and in real time.
[0003] In this demand context, intelligent data processing and application algorithms based on technologies such as deep learning are important means to solve the above problems. The training and optimization of intelligent algorithms require a large amount of data to drive them. However, due to the changeable conditions of remote sensing imaging and the limited number, types, and patterns of remote sensing images containing valuable targets, it is difficult to provide sufficient data support for intelligent algorithms. Generating remote sensing images through remote sensing scene simulation, image synthesis and other means has become an important technical direction.
[0004] Remote sensing scene simulation technology is based on simulation software and full-link modeling to simulate the generation of remote sensing images, but there are problems such as slow generation speed and low overall image authenticity; while image synthesis technology uses image cropping, geometric transformation, image fusion and image generation to achieve the purpose of acquiring target materials, image augmentation, and adding and modifying image information. These methods can achieve the generation of simulated images to a certain extent and provide training data for intelligent algorithms.
[0005] However, the simulations generated by the above methods are difficult to guarantee the generation speed, the scale of generated images, the authenticity of image targets, the controllability of information generation and other factors at the same time, and cannot be actually applied to the real-time generation tasks of remote sensing target images that require large-scale and diversified information. Summary of the invention
[0006] The purpose of the present invention is to overcome the shortcomings of the prior art and provide an information-controllable remote sensing target image simulation generation method, which realizes the generation of high-precision remote sensing target images with full process automation, reasonable target positions and controllable information.
[0007] To achieve the above-mentioned object of the invention, the present invention provides an information-controllable remote sensing target image simulation generation method, characterized in that it includes the following steps:
[0008] (1) Collect and synthesize materials;
[0009] Use 3D point cloud scanning technology to scan the target's outline to obtain the target's 3D model, and then perform rendering material, rendering texture, and rendering lighting conditions on the target's 3D model in sequence to generate N 2D images of the target at different angles as synthetic materials;
[0010] (2) Collect and process background materials;
[0011] (2.1) Select a real remote sensing image Bg as the background material of the synthetic image;
[0012] (2.2) Set the number m of mask images to be generated and set a mask image set for storing the generated mask images;
[0013] (2.3) Use the semantic segmentation model to process the background material and generate a mask image Bm containing the background type pixel by pixel, where each pixel value in Bm represents the background type;
[0014] (2.4) Randomly select a synthetic material; randomly select a pixel point on the background material as the starting point, and then place the central pixel point of the synthetic material to coincide with the starting point;
[0015] After the placement is completed, the background type of each pixel in the mask image Bm is used to automatically determine whether the target in the synthesized material meets the generation rules on the background material. If it meets the generation rules, go to step (2.5); otherwise, reselect a new starting point and continue verification until a starting point that meets the conditions is found;
[0016] (2.5) Based on the starting point in (2.4), a binary mask image Mn of the target synthesis position with the same size as the background material is generated;
[0017] (2.6) Traverse each mask image in the mask image set, and determine whether the newly generated binary mask image Mn has pixel overlap with each mask image in the mask image set. If there is no pixel overlap between the binary mask image Mn and each mask image in the mask image set, then add the newly generated binary mask image Mn to the mask image set; otherwise, scale and rotate the binary mask image Mn, and then determine whether there is pixel overlap. If there is no pixel overlap between the adjusted binary mask image Mn and each mask image in the mask image set, then add the adjusted binary mask image Mn to the mask image set; if there is still pixel overlap after the adjustment times reaches the preset maximum times, then discard the binary mask image Mn;
[0018] (2.7) Determine whether the number of mask images in the mask image set reaches m. If it reaches m, proceed to step (3); otherwise, return to step (2.4) and continue to generate mask images;
[0019] (3) Remote sensing target synthesis;
[0020] (3.1) Input N synthetic materials into the feature extractor in sequence to obtain the target feature information ID tokens;
[0021] (3.2) Extract the high-frequency feature map of the target:
[0022] Randomly select a mask image from the mask image set, find out the synthetic material corresponding to the mask image when it is generated, and record it as I erode , and then according to the synthesis material I erode The mask image is cropped to the same size as the synthesized material, and the mask image is denoted as M erode ;
[0023] Extract the high-frequency feature map of the target through a high-pass filter I h ;
[0024]
[0025] Among them, K h and K v Represent the horizontal and vertical Sobel convolution kernels, respectively, as high-pass filters;
[0026] (3.3), according to the mask map M erode Find the mask area in the background material, and then use the high-frequency feature map I h Replace the pixels in the mask area with the pixels of to obtain the spliced image Bb;
[0027] (3.4) Input the spliced image Bb into the UNet-based encoder for detail extraction to obtain multi-layer detail features with different resolutions;
[0028] (3.5) Input feature information ID tokens and multi-layer detail features into a pre-trained text-image diffusion model to output the remote sensing image after target generation;
[0029] (3.6) Taking the target remote sensing image generated in step (3.5) as the reference, traverse each mask map in the mask map set, and then follow steps (3.2)-(3.5) to add the targets in each mask map to the remote sensing image in turn, so as to obtain a multi-target remote sensing image.
[0030] The object of the invention of the present invention is achieved in this way:
[0031] The present invention is a remote sensing target image simulation generation method with fully automatic generation process, diversified generation targets and controllable information. First, synthetic materials and background materials are collected, and then the synthetic materials and background materials are processed to generate multiple mask images to obtain the target generation position; feature extraction and detail extraction are performed on the target material through an intelligent algorithm, and finally the target is generated in the background based on the mask image combined with the target feature information and detail information to obtain a remote sensing image containing the target.
[0032] At the same time, the information-controllable remote sensing target image simulation generation method of the present invention also has the following beneficial effects:
[0033] (1) The present invention can realize autonomous selection of target generation position, couple the autonomous target position generation algorithm with the subsequent target generation algorithm, and generate a high-precision simulated target with real features and consistent with the background at the selected position, thereby improving the full process autonomy and generation efficiency of remote sensing target image simulation;
[0034] (2) By carrying out high-precision three-dimensional modeling of the target, it is possible to generate high-fidelity target materials with multiple angles, multiple postures, and variable materials, providing diversified feature learning data for the target's feature learning algorithm;
[0035] (3) Through intelligent algorithms, the key features and detail features of the target material are learned separately, and while ensuring the authenticity of the key features of the target, target images with diverse postures and harmonious target background are generated. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 This is a flow chart of a method for simulating and generating remote sensing target images with controllable information in the present invention;
[0037] Figure 2 It is the background material semantic segmentation mask map;
[0038] Figure 3 This is a visualization diagram of different embedding methods;
[0039] Figure 4 An example of a multi-target remote sensing image generated. DETAILED DESCRIPTION
[0040] The specific implementation of the present invention is described below in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.
[0041] Example
[0042] Figure 1 The present invention is a flow chart of an information-controllable remote sensing target image simulation generation method.
[0043] In this embodiment, if Figure 1 As shown, the present invention provides an information-controllable remote sensing target image simulation generation method, comprising the following steps:
[0044] (1) Collect and synthesize materials;
[0045] The ship target is scanned by using 3D point cloud scanning technology to obtain a 3D model of the target, and then the 3D model of the target is rendered with materials, textures, and lighting conditions, generating 100 2D images of the target at different angles as synthetic materials. In this embodiment, the target can be selected as needed, such as an airplane, a car, etc.
[0046] (2) Collect and process background materials;
[0047] (2.1) Select a real ocean remote sensing image Bg as the background material of the synthetic image;
[0048] In this embodiment, after having the synthesis material and the background material, for each type of foreground object to be fused, we need to set an area where it should originally exist, such as ship-ocean, airplane-airport, car-road, etc.
[0049] (2.2) Set the number of mask images to be generated to 10, and set a mask image set to store the generated mask images;
[0050] (2.3) Use the semantic segmentation model to process the background material and generate a mask image Bm containing the background type pixel by pixel, where each pixel value in Bm represents the background type;
[0051] In this embodiment, we use the semantic segmentation model FastFCN to process the provided background image to accurately segment different objects in the image, including oceans, airports, roads, ships, airplanes, cars, etc. This step is crucial for subsequent target addition because it helps us understand and define the semantic information of each part in the image.
[0052] In this embodiment, the synthesis material takes a ship as an example, and the background material takes the ocean as an example. Figure 2 As shown, the generated mask map is located in the specified ocean area.
[0053] After obtaining accurate mask information, we can theoretically use graph transformation and cropping techniques to embed the target into the aerial image. Figure 3 As shown, Figure 3 The foreground object shown in (a) is pasted into Figure 3 (b) shows the background. This often results in an unnatural fusion effect between the target and the surrounding environment, showing a clear boundary and sense of separation, such as Figure 3 (c) This unrealistic data will undoubtedly have a negative impact on the performance of subsequent remote sensing target detection or semantic segmentation models, so we need to perform the following steps to obtain Figure 3 (d) shows the effect diagram, and the specific processing steps are as follows:
[0054] (2.4) Randomly select a synthetic material; randomly select a pixel point on the background material as the starting point, and then place the central pixel point of the synthetic material to coincide with the starting point;
[0055] After the placement is completed, the background type of each pixel in the mask image Bm is used to automatically determine whether the target in the synthesized material meets the generation rules on the background material. If it meets the generation rules, go to step (2.5); otherwise, reselect a new starting point and continue verification until a starting point that meets the conditions is found;
[0056] In this embodiment, it is determined whether the starting point is located in the semantic segmentation area corresponding to the target object. For example, if the target is a ship, the starting point must be located in the ocean area; if the target is an airplane, the starting point must be located in the airport area; similarly, if the target is a car, the starting point should be located in the road area.
[0057] (2.5) Based on the starting point in (2.4), a binary mask image Mn of the target synthesis position with the same size as the background material is generated;
[0058] In order to reasonably generate the target into the remote sensing image, we must carefully select and specify the appropriate area in the image. In this process, we must carefully consider and solve a key problem, that is, whether the generated mask area overlaps with the original target in the image. If the mask area overlaps with the original target, then this will cause the foreground target to be mistakenly placed in an unreasonable area in the image in the subsequent pasting operation.
[0059] (2.6) Traverse each mask image in the mask image set, and determine whether the newly generated binary mask image Mn has pixel overlap with each mask image in the mask image set. If there is no pixel overlap between the binary mask image Mn and each mask image in the mask image set, then add the newly generated binary mask image Mn to the mask image set; otherwise, scale and rotate the binary mask image Mn, and then determine whether there is pixel overlap. If there is no pixel overlap between the adjusted binary mask image Mn and each mask image in the mask image set, then add the adjusted binary mask image Mn to the mask image set; if there is still pixel overlap after the adjustment times reaches the preset maximum times, then discard the binary mask image Mn;
[0060] In this embodiment, when the binary mask image Mn is scaled and rotated, the scaling factor generally varies in the range of 0.8 to 1.0, and the rotation angle generally varies in the range of 0 to 180 degrees;
[0061] (2.7) Determine whether the number of mask images in the mask image set reaches 10. If so, proceed to step (3); otherwise, return to step (2.4) and continue to generate mask images;
[0062] (3) Remote sensing target synthesis;
[0063] (3.1) Input 100 synthetic materials into the feature extractor in sequence to obtain the target’s feature information IDtokens;
[0064] (3.2) Extract the high-frequency feature map of the target:
[0065] Randomly select a mask image from the mask image set, find out the synthetic material corresponding to the mask image when it is generated, and record it as I erode , and then according to the synthesis material I erode The mask image is cropped to the same size as the synthesized material, and the mask image is denoted as M erode ;
[0066] Extract the high-frequency feature map of the target through a high-pass filter I h ;
[0067]
[0068] Among them, K h and K v Represent the horizontal and vertical Sobel convolution kernels, respectively, as high-pass filters;
[0069] (3.3), according to the mask map M erode Find the mask area in the background material, and then use the high-frequency feature map I h Replace the pixels in the mask area with the pixels of to obtain the spliced image Bb;
[0070] (3.4) Input the spliced image Bb into a detail extractor for detail extraction to obtain multi-layer detail features with different resolutions; in this embodiment, the detail extractor is an encoder based on UNet;
[0071] (3.5) Input feature information ID tokens and multi-layer detail features into a pre-trained text-image diffusion model to output the remote sensing image after target generation;
[0072] (3.6) Taking the target remote sensing image generated in step (3.5) as the reference, traverse each mask image in the mask image set, and then follow steps (3.2)-(3.5) to add the target in each mask image to the remote sensing image in turn, so as to obtain a multi-target remote sensing image, such as Figure 4 shown.
[0073] Although the above describes the illustrative specific embodiments of the present invention to facilitate those skilled in the art to understand the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.
Claims
1. A method for generating a remote sensing target image with controllable information, characterized in that: The following steps are involved: (1) Collect and synthesize materials; Use 3D point cloud scanning technology to scan the target's outline to obtain the target's 3D model, and then perform rendering material, rendering texture, and rendering lighting conditions on the target's 3D model in sequence to generate N 2D images of the target at different angles as synthetic materials; (2) Collect and process background materials; (2.1) Select a real remote sensing image Bg as the background material of the synthetic image; (2.2) Set the number m of mask images to be generated and set a mask image set for storing the generated mask images; (2.3) Use the semantic segmentation model to process the background material and generate a mask image Bm containing the background type pixel by pixel, where each pixel value in Bm represents the background type; (2.4) Randomly select a synthetic material; randomly select a pixel point on the background material as the starting point, and then place the central pixel point of the synthetic material to coincide with the starting point; After the placement is completed, the background type of each pixel in the mask image Bm is used to automatically determine whether the target in the synthesized material meets the generation rules on the background material. If it meets the generation rules, go to step (2.5); otherwise, reselect a new starting point and continue verification until a starting point that meets the conditions is found; (2.5) Based on the starting point in (2.4), a binary mask image Mn of the target synthesis position with the same size as the background material is generated; (2.6) Traverse each mask image in the mask image set, and determine whether the newly generated binary mask image Mn has pixel overlap with each mask image in the mask image set. If there is no pixel overlap between the binary mask image Mn and each mask image in the mask image set, then add the newly generated binary mask image Mn to the mask image set; otherwise, scale and rotate the binary mask image Mn, and then determine whether there is pixel overlap. If there is no pixel overlap between the adjusted binary mask image Mn and each mask image in the mask image set, then add the adjusted binary mask image Mn to the mask image set; if there is still pixel overlap after the adjustment times reaches the preset maximum times, then discard the binary mask image Mn; (2.7) Determine whether the number of mask images in the mask image set reaches m. If it reaches m, proceed to step (3); otherwise, return to step (2.4) and continue to generate mask images; (3) Remote sensing target synthesis; (3.1) Input N synthetic materials into the feature extractor in sequence to obtain the target feature information ID tokens; (3.2) Extract the high-frequency feature map of the target: Randomly select a mask image from the mask image set, find out the synthetic material corresponding to the mask image when it is generated, and record it as I erode , and then according to the synthesis material I erode The mask image is cropped to the same size as the synthesized material, and the mask image is denoted as M erode ; Extract the high-frequency feature map of the target through a high-pass filter I h ; Among them, K h and K v Represent the horizontal and vertical Sobel convolution kernels, respectively, as high-pass filters; (3.3), according to the mask map M erode Find the mask area in the background material, and then use the high-frequency feature map I h Replace the pixels in the mask area with the pixels of to obtain the spliced image Bb; (3.4) Input the spliced image Bb into the UNet-based encoder for detail extraction to obtain multi-layer detail features with different resolutions; (3.5) Input feature information ID tokens and multi-layer detail features into a pre-trained text-image diffusion model to output the target remote sensing image; (3.6) Taking the target remote sensing image generated in step (3.5) as the reference, traverse each mask map in the mask map set, and then follow steps (3.2)-(3.5) to add the targets in each mask map to the remote sensing image in turn, so as to obtain a multi-target remote sensing image.
