An image harmonization method and device based on low-altitude simulation demand and a medium

Abstract

The application provides an image harmonization method based on low-altitude simulation requirements, comprising the following steps: acquiring an aerial picture, which is defined as a first image; performing disharmonization processing on the first image to generate a second image with disharmonization factors and a foreground mask; providing a harmonization network model to generate a harmonization prediction image, which is defined as a third image; comparing the first image and the third image to obtain an error, and optimizing the harmonization network model until the third image approaches the first image. The application does not need to collect a large number of real aerial pictures, but only needs to collect a small number of real aerial pictures to train and use the harmonization network model, thereby realizing the realization requirement of simulation pictures.

Description

Technical Field

[0001] This invention relates to the field of scene simulation technology, and in particular to an image harmonization method, device and medium based on low-altitude simulation requirements. Background Technology

[0002] With the development of the low-altitude economy, related products in industries such as tourism, security patrols, medical rescue, aerial firefighting, agricultural and forestry protection, and express logistics are increasingly demanding simulations of aerial scenarios. For example, eVTOL (electric vertical takeoff and landing) aircraft or drones cannot be directly tested in the air during the research and development and testing phase; they require extensive testing through simulations of real aerial scenarios. Therefore, the construction of low-altitude environmental images and scenarios is essential.

[0003] The existing technology includes the following three methods for processing:

[0004] 1. Real-world data acquisition primarily focuses on ground-based imagery and scene construction, with applications in fields such as autonomous vehicles. Ground-based scenes are relatively easy to acquire, and the abundance of image elements makes it easy to capture various complex situations, facilitating system acquisition, recognition, and simulation optimization. However, capturing complex low-altitude scenes is challenging. It typically requires a long shooting time to obtain aerial images of aircraft or drones, and the resulting variations in attitudes and positional relationships often fail to meet the needs of simulation testing. Furthermore, acquiring real aerial data is very costly.

[0005] 2. Simulator Generation. Existing simulators include AIRSIM and ANSYS. AIRSIM lacks scene richness and detail, and requires numerous sensors; ANSYS has high modeling costs and is difficult to use, and the technology is still immature. Furthermore, the simulated scenes still have inconsistencies and cannot meet the needs of actual low-altitude scene testing.

[0006] 3. Algorithm generation. The algorithm's cost and the simulator's setup present inconsistencies, making it impossible to directly generate harmonious scenes that closely resemble real images through simple processing, thus failing to meet practical needs. Summary of the Invention

[0007] This invention provides an image harmonization method based on low-altitude simulation requirements to address the issues of high acquisition costs and difficulty in meeting testing needs.

[0008] This invention provides an image harmonization method based on low-altitude simulation requirements, comprising the following steps:

[0009] S1: Collect real aerial images, defined as the first image, which includes a low-altitude background and relatively prominent foreground objects;

[0010] S2: Perform incongruity processing on the first image to generate a second image with incongruity elements and a foreground mask;

[0011] S3: Provide a harmonization network model, input the second image and the foreground mask into the harmonization network model, generate a harmonization prediction map, and define it as the third image;

[0012] S4: Compare the error between the first image and the third image, and feed the error back to the harmonization network model to correct the third image;

[0013] S5: Repeat steps S3 and S4 to optimize the harmonization network model until a third image that closely resembles the first image is generated.

[0014] Specifically, the image harmonization method based on low-altitude simulation requirements also includes:

[0015] S6: Based on the optimized harmonization network model, input the simulated aerial image, which is defined as a simulated image. After processing by the harmonization network model, the simulated image generates an aerial image that is close to the real one.

[0016] Specifically, real aerial images are collected, and the first image with the foreground object and the pure background image are selected. The first image is used to optimize the harmonization network model. After optimization, the pure background image and the foreground mask are used by the simulator or algorithm to generate the simulated image with scene and category changes. After being processed by the harmonization network model, a real image or a changing aerial scene is generated.

[0017] Specifically, S2 further includes:

[0018] S21: Label the foreground objects in the first image;

[0019] S22: Extract the marked foreground objects, perform indecorative processing on the foreground objects, generate a simulated foreground and generate a foreground mask;

[0020] S23: After the foreground object is extracted from the first image, it is repaired to generate a background image;

[0021] S24: Combine the simulated foreground image with the background image to generate the second image.

[0022] Specifically, in S6 and / or S24, the MPB algorithm is used to synthesize the simulated foreground and the background image into the second image or the simulated image.

[0023] Specifically, in step S21, the foreground objects in the first image are labeled. This is done simultaneously when the aerial image is acquired in step S1. Based on the real or simulated IMU and GPS, an algorithm is used to label the foreground objects in the acquired first image.

[0024] Specifically, the harmonization network model includes an encoder and a decoder, and corresponding to step S3, it further includes:

[0025] S31: Provide the encoder, receive the second image and the foreground mask, and encode to generate image features;

[0026] S32: Provide the decoder, receive the image features, and generate a harmonized prediction map, i.e., the third image.

[0027] Specifically, S4 includes:

[0028] S41: Calculate the error between the first image and the second image using a loss function;

[0029] S42: Based on the error, optimize the harmonized network model using the backpropagation algorithm.

[0030] Specifically, in S1, the foreground object is one or more of the following: white clouds, airplanes, or obstacles appearing at low altitudes.

[0031] The present invention provides a computer device, the computer device including a memory and a processor coupled to the memory; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the image harmonization method based on low-altitude simulation requirements.

[0032] The present invention provides a computer-readable storage medium for storing a computer program, which, when executed by a computer, is used to implement the image harmonization method based on low-altitude simulation requirements.

[0033] Compared with existing technologies, the image harmonization method based on low-altitude simulation requirements provided by this invention only requires a small number of real aerial images. The first image that meets the requirements is preprocessed, and based on the training of the harmonization network model, a third image close to the real aerial image can be generated. This method does not require collecting a large number of real aerial images; it only needs to collect a small number of real aerial images to train the harmonization network model, thereby achieving the realism requirement of the simulated image. When different low-altitude scenes need to be simulated, foreground objects can be generated through a simulator or algorithm, and then combined with a real low-altitude background to synthesize a near-realistic simulated image or scene.

[0034] The image harmonization method based on low-altitude simulation requirements described above can not only meet the needs of eVTOL or UAVs in the current research and development testing phase in the low-altitude field by providing a large number of harmonized simulation scenarios to achieve realistic and effective testing, but also greatly save the time and cost of collecting real aviation data. It also eliminates the need to use costly and difficult-to-use simulators. The image harmonization method based on low-altitude simulation requirements provided in this application only requires users to provide simple simulation images or scenarios to achieve rich and realistic harmonized images or scenarios. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein:

[0036] Figure 1 This is a flowchart of the image harmonization method based on low-altitude simulation requirements provided by the present invention;

[0037] Figure 2 This is a structural block diagram of the image harmonization method based on low-altitude simulation requirements provided by the present invention;

[0038] Figure 3 This is a schematic diagram of the first image acquired in one embodiment of the image harmonization method based on low-altitude simulation requirements provided by the present invention;

[0039] Figure 4 yes Figure 1 The flowchart of S2 shown below;

[0040] Figure 5 This is a schematic diagram of the background image generated after processing the first image in one embodiment;

[0041] Figure 6 This is a schematic diagram of the simulated foreground generated after processing the first image in one embodiment;

[0042] Figure 7 This is a schematic diagram of the second image generated after processing the first image in one embodiment;

[0043] Figure 8 yes Figure 1 The flowchart of S3 shown is shown below;

[0044] Figure 9 This is a schematic diagram of a device according to an embodiment of the present invention;

[0045] Figure 10 This is a schematic diagram of the structure of a computer-readable storage medium according to an embodiment of the present invention. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0047] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.

[0048] Furthermore, in this invention, descriptions involving "first," "second," etc., are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0049] Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but only if they are feasible for those skilled in the art. If the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0050] Please see Figure 1 and Figure 2 , Figure 1 This is a flowchart of the image harmonization method based on low-altitude simulation requirements provided by the present invention. Figure 2 This is a structural block diagram of the image harmonization method based on low-altitude simulation requirements provided by the present invention. The image harmonization method based on low-altitude simulation requirements includes the following steps:

[0051] S1: Acquire aerial images, defined as first image 1, which includes a low-altitude background and relatively prominent foreground objects;

[0052] S2: Perform incongruity processing on the first image 1 to generate a second image 3 with incongruity elements and a foreground mask 15;

[0053] S3: Provide a harmonization network model 5, input the second image 3 and the foreground mask 15 into the harmonization network model 5, generate a harmonization prediction map, and define it as the third image 7;

[0054] S4: Compare the error between the first image 1 and the third image 7, and feed the error back to the harmonization network model 5 to correct the third image 7;

[0055] S5: Repeat steps S3 and S4 to optimize the harmonization network model 5 until the third image 7, which is close to the first image 1, is generated.

[0056] S6: Based on the optimized harmonization network model 5, input the simulated aerial image, which is defined as a simulated image. After processing by the harmonization network model 5, the simulated image generates an aerial image that is close to the real one.

[0057] Please continue to combine Figures 3 to 7 The image harmonization method based on low-altitude simulation requirements is further explained below. Figure 3 The first image 1 is obtained by capturing real aerial images. It should be noted that in step S1, the foreground object is one or more of the following: clouds, an airplane, or obstacles appearing at low altitude. Correspondingly, in this embodiment, the first image 1 includes the low-altitude background and the foreground object, which is an airplane; however, in other embodiments, it can be an eVTOL or drone of various types and sizes, or it can be an obstacle such as a balloon, a bird, or a high-altitude crane.

[0058] Furthermore, in order to train a more intelligent and comprehensive harmonization network model 5 and ensure that it can generate simulated images or scenes that are closer to reality, during the process of acquiring the aerial images, it is necessary to acquire as many first images 1 containing various foreground objects as possible, and filter out images without any foreground objects, so as to avoid the image harmonization method based on low-altitude simulation requirements processing too much invalid data and reduce training costs.

[0059] It should be noted that there are various methods for acquiring the aerial images. For example, real aerial images can be collected using various existing low-altitude drones, or virtual aerial images can be generated using simulators or algorithms. Since the training of the harmonization network model 5 mainly provides the first image 1 and the third image 7 for comparison, the aerial images do not need to be completely realistic. Therefore, multiple methods can be combined to acquire the aerial images, reducing training costs.

[0060] To facilitate the construction of more realistic simulated images, it is necessary to extract the foreground objects for conversion and matching to construct the simulated images. For example... Figure 4 As shown, step S2 further includes the following steps:

[0061] S21: Label the foreground objects in the first image 1;

[0062] S22: Extract the marked foreground objects, perform indecorative processing on the foreground objects, generate simulated foreground 13 and generate foreground mask 15;

[0063] S23: After the foreground object is extracted from the first image 1, it is repaired to generate the background image 11;

[0064] S24: Combine the simulated foreground 13 with the background image 11 to generate the second image 3.

[0065] In this embodiment, the airplane is marked in the first image 1, and the airplane is extracted from the first image 1, which is equivalent to splitting the first image 1 into an image with only foreground and an image with only background. The image with only background, lacking the airplane, needs further restoration to generate the background image 11, as shown below. Figure 5 As shown. The background image 11, which originally depicted the area where the aircraft was located, has been processed to blend it with the surrounding environment and form a complete image.

[0066] The proposed method involves deconstructing the aircraft image to generate a simulated foreground 13 and a foreground mask 15. For example... Figure 6 As shown, except for the aircraft, the surrounding area is blacked out to form an image of the same size as the first image 1, containing only the foreground object. Image masking is a common technique in image processing, mainly used to identify or limit specific areas in an image that need to be processed. In S22, image masking is used to process the foreground object, facilitating the use of deep learning in the harmonization network model 5. By inputting the mask, the image data can be denoised.

[0067] Furthermore, applying image masking techniques to the processing of the foreground objects also facilitates the generation of the simulated images used for subsequent testing. After the harmonization network model 5 is trained, a large number of simulated images are needed through a simulator or algorithm. During the generation of the simulated images, the foreground mask 15 can be provided and paired with various backgrounds, or multiple foreground masks 15 can be combined to provide more and richer test scenarios.

[0068] To train the harmonized network model 5, please refer to further documentation. Figure 7The simulated foreground 13 is combined with the background image 11 to generate the second image 3, as shown below. Figure 7 As shown. Since the training of the harmonization network model 5 requires image data, by synthesizing the simulated foreground 13 and the background image 11, a preliminary simulated second image 3 can be obtained. The first image 1 and the second image 3 form a pair of harmonious and disharmonious image relationships, which can drive the learning and training of the harmonization network model 5, enabling it to ultimately output a simulated image that approaches reality. In this embodiment, by providing the first image 1 and performing disharmonization processing on it, a disharmonious second image 3 for comparison can be obtained. This eliminates the need for simulators or algorithm generation, simplifying the operation, reducing algorithmic requirements, and lowering the overall training cost of the harmonization network model 5.

[0069] It should be noted that various algorithms can be used to synthesize the simulated foreground 13 and the background image 11, such as Alpha fusion, pyramid fusion, or Poisson fusion. In this embodiment, the MPB algorithm is preferred. The main contents of the MPB algorithm are summarized as follows:

[0070] 1. Input parameters

[0071] The system provides a source image S (the simulated foreground 13, the object region to be fused), a target image T (the background image 11), and a binary mask Ω. The binary mask Ω represents the fusion region of S within T, and Ω=1 represents the region to be fused.

[0072] 2. Gradient field calculation

[0073] Define the gradient of the source image: ∇S=( , );

[0074] Define the gradient of the target image: ∇T=( , );

[0075] Define a weighted mixed gradient: G = α∇S + (1−α)∇T; where α is the dynamic weight.

[0076] 3. Construct the Poisson equation

[0077] Construct the Poisson equation: ∇ 2 Φ = div(G)inΩ; where Φ is the fused image to be solved, ∇ 2 is the Laplace operator, and div is the divergence.

[0078] Define boundary conditions: Φ∣= T∣ (the pixel value of the fusion region boundary is equal to the target image).

[0079] 4. Discretization and Solution

[0080] The Laplace operator ∇ 2 Discretize into a five-point template:

[0081] ∇ 2 Φ≈Φ(i+1,j)+Φ(i−1,j)+Φ(i,j+1)+Φ(i,j−1)−4Φ(i,j);

[0082] The divergence term div(G) is discretized as: div(G) ≈ G x (i+1,j)−G x (i,j)+G y (i,j+1)−G y (i,j);

[0083] The Poisson equation can be constructed as a linear system: AΦ=b.

[0084] Where A is a sparse symmetric matrix (each pixel corresponds to an equation), b is div(G) and boundary condition filling; iterative methods (such as conjugate gradient method, multigrid method) or direct methods (for small-scale problems) are used.

[0085] 5. The output is a seamlessly blended image R(p), where p is inside Ω and Φ(p) is used, and p is outside Ω and T(p) is used.

[0086] The MPB algorithm employs parameter input, gradient field calculation, component Poisson equation, discretization, and solution steps to ultimately output a seamlessly fused image. This not only meets the requirements for generating the second image 3, but more importantly, it facilitates the synthesis of a large number of simulated images after the harmonization network model 5 has been trained. In other words, in S6 and / or S24, the MPB algorithm can be used to synthesize the simulated foreground 13 and the background image 11 into the second image 3 or the simulated image. The simulated image generated in S6, when input into the harmonization network model 5, can also quickly respond to recognition and process to generate near-realistic aerial images. Moreover, the background image 11 mainly contains images of a large amount of blue sky; using the MPB algorithm to fuse the simulated foreground 13 and the background image 11 can effectively improve fusion efficiency without requiring overly complex algorithms, further saving model training costs.

[0087] In S1, real aerial images are acquired and further filtered, classified, and used. The first image 1 with the foreground object and the pure background image are filtered out. The first image 1 is used to optimize the harmonization network model 5. After optimization, the pure background image and the foreground mask 15 are used by the simulator or algorithm to generate simulated images of multiple scenes and categories. After being processed by the harmonization network model 5, realistic images or changing aerial scenes are generated.

[0088] By filtering and classifying, not only can the collected real aerial image data be accurately applied, ensuring the accuracy of model training, but the first image 1 and the pure background image selected through filtering and classification can also be effectively used, reducing acquisition costs. After the harmonization network model 5 is optimized, the simulated images with a large number of scenes are generated. Moreover, based on the simulated images with a large number of scenes, changing aerial scenes can be further generated, increasing the items that can be tested in the simulation testing phase, including static parameter testing and dynamic obstacle avoidance.

[0089] It should be noted that the changing aerial scene includes not only aerial scenes in which the foreground object moves, flips, or changes direction relative to the background image, but also aerial scenes in which background lighting, color changes, and image style conversions change.

[0090] Please continue to refer to the following: Figure 1 and Figure 8 The harmonization network model 5 includes an encoder 51 and a decoder 53, and corresponding to step S3, it further includes:

[0091] S31: The encoder 51 is provided to receive the second image 3 and the foreground mask 15, and to encode and generate image features 55;

[0092] S32: Provide the decoder 53 to receive the image features 55 and generate a harmonized prediction map, i.e., the third image 7.

[0093] The harmonization network model 5 can be constructed through the encoder 51 and the decoder 53 to process the first image 1 and the foreground mask 15, thereby generating the harmonization prediction map.

[0094] The encoder 51 can select a general feature extraction network to extract image features 55 at different feature dimensions. For example, network models such as Transformer, Unet, ShuffleNet, or RRDBnet can be used as the encoder 51. The image features 55 can represent, for example, features of the foreground object or features of the background. Different feature dimensions can include texture, brightness, and color features, and extraction can be performed at multiple different resolutions. This allows for a comprehensive and accurate understanding of the details and overall information of the second image 3 and the foreground mask 15, which is beneficial for improving the realism of the harmonized prediction image.

[0095] The decoder 53 can select a general image generation network, such as using ResNet, Transformer combined with downsampling or deconvolution, to decode and make full use of the features encoded by the encoder 51 to restore the harmonized prediction image.

[0096] It should be noted that the specific network construction and algorithm of the encoder 51 and the decoder 53 can be selected based on the needs, and will not be described in detail here.

[0097] To optimize the harmonization network model 5, the error between the first image 1 and the third image 7 is compared and fed back to the harmonization network model 5 to correct the third image 7. Step S4 further includes the following steps:

[0098] S41: Calculate the error between the first image 1 and the second image 3 using a loss function;

[0099] S42: Based on the error, optimize the harmonization network model 5 using the backpropagation algorithm.

[0100] The loss function, also known as the error function, is a function used to measure the difference between the predicted result and the actual result. It evaluates the performance of the harmonization network model 5 by calculating the degree of inconsistency between the predicted value of the first image 1 and the actual value of the third image 7. Of course, the loss function can use the mean squared error (MSE) loss function, the mean absolute error (MAE) loss function, or the Huber loss function, etc., to calculate the error; further explanation is not provided here.

[0101] The backpropagation algorithm, combined with the loss function, is used to train the harmonization network model 5. It primarily calculates and updates the parameters of the harmonization network model 5 to minimize the loss between the predicted output and the actual target. Within the deep learning framework, the harmonization network model 5 consists of multiple layers, each containing multiple nodes (or neurons). The error is propagated back from the output layer along the network layers, and the weights of each layer are adjusted based on the propagated error signals, gradually improving the network's learning ability on the input data. Steps S3 and S4 are repeated, allowing the harmonization network model 5 to undergo multiple iterations of training until it converges or reaches a predetermined number of training rounds.

[0102] The error is equivalent to the image feature 55 extracted by the encoder 51. Based on the feedback of the error, the harmonization network model 5 can be continuously optimized through the loss function and the backpropagation algorithm.

[0103] Of course, in step S5, a certain error threshold can be set relative to the image feature 55. When the error meets the preset threshold range, the training of the harmonization network model 5 can be stopped, and the trained model can be stored. Specific threshold settings are not limited here.

[0104] Furthermore, during the training process, the image harmonization method based on low-altitude simulation requirements also includes repeating S1-S5, and by providing the first image 1 from multiple scenes, the harmonization network model 5 is trained in multiple layers to ensure that the generation of the simulated image meets the requirements.

[0105] Compared with existing technologies, the image harmonization method based on low-altitude simulation requirements provided by this invention only requires a small number of real aerial images. The first image 1, which meets the requirements, is preprocessed, and based on the training of the harmonization network model 5, a third image 7 that closely resembles a real aerial image can be generated. This method does not require collecting a large number of real aerial images; it only requires collecting a small number of real aerial images to train the harmonization network model 5, thereby achieving the realism requirement for simulated images. When different low-altitude scenes need to be simulated, foreground objects can be generated through a simulator or algorithm, and then combined with a realistic low-altitude background to synthesize a near-realistic simulated image or scene.

[0106] The image harmonization method based on low-altitude simulation requirements described above can not only meet the needs of eVTOL or UAVs in the current research and development testing phase in the low-altitude field by providing a large number of harmonized simulation scenarios to achieve realistic and effective testing, but also greatly save the time and cost of collecting real aviation data. It also eliminates the need to use costly and difficult-to-use simulators. The image harmonization method based on low-altitude simulation requirements provided in this application only requires users to provide simple simulation images or scenarios to achieve rich and realistic harmonized images or scenarios.

[0107] Please continue reading. Figure 9 In one embodiment, a computer device 100 is provided, the computer device 100 including a memory 101 and a processor 103 coupled to the memory 101. The memory 101 is used to store program data, and the processor 103 is used to execute the program data. The program data includes the following steps:

[0108] S1: Collect real aerial images, defined as first image 1, which includes a low-altitude background and relatively prominent foreground objects;

[0109] S2: Perform incongruity processing on the first image 1 to generate a second image 3 with incongruity elements and a foreground mask 15;

[0110] S3: Provide a harmonization network model 5, input the second image 3 and the foreground mask 15 into the harmonization network model 5, generate a harmonization prediction map, and define it as the third image 7;

[0111] S4: Compare the error between the first image 1 and the third image 7, and feed the error back to the harmonization network model 5 to correct the third image 7;

[0112] S5: Repeat steps S3 and S4 to optimize the harmonization network model 5 until the third image 7, which is close to the first image 1, is generated.

[0113] Please continue reading. Figure 9 In one embodiment, a computer-readable storage medium 300 is provided, on which a computer program 301 is stored, which, when executed, performs the following steps:

[0114] S1: Collect real aerial images, defined as first image 1, which includes a low-altitude background and relatively prominent foreground objects;

[0115] S2: Perform incongruity processing on the first image 1 to generate a second image 3 with incongruity elements and a foreground mask 15;

[0116] S3: Provide a harmonization network model 5, input the second image 3 and the foreground mask 15 into the harmonization network model 5, generate a harmonization prediction map, and define it as the third image 7;

[0117] S4: Compare the error between the first image 1 and the third image 7, and feed the error back to the harmonization network model 5 to correct the third image 7;

[0118] S5: Repeat steps S3 and S4 to optimize the harmonization network model 5 until the third image 7, which is close to the first image 1, is generated.

[0119] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0120] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0121] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. An image harmonization method based on low-altitude simulation requirements, characterized in that, Includes the following steps: S1: Acquire aerial images, defined as the first image, which includes a low-altitude background and relatively prominent foreground objects; S2: Perform incongruity processing on the first image to generate a second image with incongruity elements and a foreground mask; S3: Provide a harmonization network model, input the second image and the foreground mask into the harmonization network model, generate a harmonization prediction map, and define it as the third image; S4: Compare the error between the first image and the third image, and feed the error back to the harmonization network model to correct the third image; S5: Repeat steps S3 and S4 to optimize the harmonization network model until a third image that is close to the first image is generated; S2 further includes: S21: Label the foreground objects in the first image; S22: Extract the marked foreground objects, perform indecorative processing on the foreground objects, generate a simulated foreground and generate a foreground mask; S23: After the foreground object is extracted from the first image, it is repaired to generate a background image; S24: Combine the simulated foreground image with the background image to generate the second image; The harmonization network model includes an encoder and a decoder, and corresponding to step S3, it also includes: S31: Provide the encoder, receive the second image and the foreground mask, and encode to generate image features; S32: Provide the decoder to receive the image features and generate a harmonized prediction map, i.e., the third image; S4 includes: S41: Calculate the error between the first image and the second image using a loss function; S42: Based on the error, optimize the harmonized network model using the backpropagation algorithm.

2. The image harmonization method based on low-altitude simulation requirements according to claim 1, characterized in that, Also includes: S6: Based on the optimized harmonization network model, input the simulated aerial image, which is defined as a simulated image. After processing by the harmonization network model, the simulated image generates an aerial image that is close to the real one.

3. The image harmonization method based on low-altitude simulation requirements according to claim 2, characterized in that, Real aerial images are collected, and the first image with the foreground object and the pure background image are selected. The first image is used to optimize the harmonization network model. After optimization, the pure background image and the foreground mask are used by the simulator or algorithm to generate the simulated image with scene and category changes. After being processed by the harmonization network model, a real image or a changing aerial scene is generated.

4. The image harmonization method based on low-altitude simulation requirements according to claim 3, characterized in that, In S6 and / or S24, the MPB algorithm is used to synthesize the simulated foreground and the background image into the second image or the simulated image.

5. The image harmonization method based on low-altitude simulation requirements according to claim 4, characterized in that, In step S21, the foreground objects in the first image are labeled simultaneously using an algorithm based on real or simulated IMU and GPS data when the aerial image is acquired in step S1.

6. A computer device, characterized in that, The computer device includes a memory and a processor coupled to the memory; The memory is used to store program data, and the processor is used to execute the program data to implement the image harmonization method based on low-altitude simulation requirements as described in any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program, which, when executed by a computer, is used to implement the image harmonization method based on low-altitude simulation requirements as described in any one of claims 1 to 5.

Images