Image harmonization method and device based on low-altitude simulation requirement and medium

Abstract

The invention provides an image harmonization method based on a low-altitude simulation demand, and the method comprises the steps: obtaining an aviation picture, and defining the aviation picture as a first image; performing disharmony processing on the first image to generate a second image with disharmony factors and a foreground mask; providing a harmonious network model, generating a harmonious prediction map, and defining the harmonious prediction map as a third image; and comparing to obtain an error between the first image and the third image, and optimizing the harmonious network model until the third image close to the first image is generated. According to the method, a large number of real aviation pictures do not need to be collected, only a small number of real aviation pictures need to be collected, the harmonious network model is trained and used, and therefore the reality requirement of the simulation pictures is met.

Description

Technical Field

[0001] The present invention relates to the technical field of scene simulation, and in particular to an image harmonization method, device and medium based on low-altitude simulation requirements. Background Art

[0002] With the development of the low-altitude economy, the demand for aerial scene simulation is increasing for products related to this sector, such as tourism, security inspections, medical rescue, aerial firefighting, agricultural and forestry protection, and express logistics. For example, during the R&D and testing phases of eVTOL (electric vertical take-off and landing) and drones, direct aerial testing is not possible. Extensive testing requires simulating real-world aerial scenarios. Therefore, the construction of low-altitude environmental images and scenarios is essential.

[0003] The existing technology includes the following three methods for processing: 1. Real-world data collection primarily focuses on ground-based imagery and scene construction, and is applied to areas such as autonomous vehicles. Because ground-based scenes are relatively simple to capture and contain numerous image elements, various complex situations are easily captured, facilitating system acquisition, identification, and simulation optimization. However, capturing these complex scenes at low altitudes is difficult. Capturing aircraft or drones in the air typically requires significant time, and capturing their various poses and positional relationships is insufficient for simulation testing. Furthermore, acquiring real-world aerial data is costly.

[0004] 2. Simulator Generation. Existing simulators include AIRSIM and ANSYS. AIRSIM's scenarios are insufficient, lacking detail, and require numerous sensors. ANSYS's modeling costs are high, the software is difficult to use, and its technology is still immature. Furthermore, the generated scenarios still suffer from inconsistencies and cannot meet the needs of actual low-altitude scenario testing.

[0005] 3. Algorithm generation: There are inconsistencies in algorithm cost and simulator construction. It is impossible to directly generate a harmonious scene close to the real image through simple processing, which makes it difficult to meet actual needs. Summary of the Invention

[0006] The present invention provides an image harmonization method based on low-altitude simulation requirements for solving the above-mentioned problems of high acquisition cost and difficulty in meeting test requirements.

[0007] The present invention provides an image harmonization method based on low-altitude simulation requirements, comprising the following steps: S1: Collect a real aerial picture, defined as a first image, which includes a low-altitude background and a relatively prominent foreground object; S2: performing a disharmony process on the first image to generate a second image with disharmony factors and a foreground mask; S3: providing a harmonization network model, inputting the second image and the foreground mask into the harmonization network model, and generating a harmonization prediction image, which is defined as a third image; S4: comparing the first image and the third image to obtain an error, feeding the error back to the harmonized network model, and correcting the third image; S5: Repeat steps S3 and S4 to optimize the harmonized network model until the third image close to the first image is generated.

[0008] Specifically, the image harmonization method based on low-altitude simulation requirements further includes: S6: Based on the optimized harmonized network model, a simulated aerial image is input, which is defined as a simulated image. The simulated image is processed by the harmonized network model to generate an aerial image close to the real one.

[0009] Specifically, real aerial pictures are collected, and the first image with the foreground object and the pure background image are screened out. The first image is used to optimize the harmonized network model. After the optimization is completed, the pure background image and the foreground mask are used in a simulator or algorithm to generate the simulated image with scene and category changes. After being processed by the harmonized network model, a real image or a changed aerial scene is generated.

[0010] Specifically, the S2 further includes: S21: marking the foreground object in the first image; S22: extracting the marked foreground object, performing a de-harmonization process on the foreground object, generating a simulated foreground and a foreground mask; S23: extracting the foreground object from the first image and performing restoration to generate a background image; S24: Synthesize the simulated foreground image and the background image to generate the second image.

[0011] Specifically, in S6 and / or S24, the MPB algorithm is used to synthesize the simulated foreground image and the background image into the second image or the simulated image.

[0012] Specifically, in the S21, the foreground object in the first image is labeled, and this is performed synchronously when the aerial picture is acquired in the S1. The foreground object is labeled in the acquired first image using an algorithm based on the real or simulated IMU and GPS.

[0013] Specifically, the harmonized network model includes an encoder and a decoder, and corresponding to step S3, further includes: S31: providing the encoder, receiving the second image and the foreground mask, and encoding to generate image features; S32: Provide the decoder, receive the image features, and generate a harmonized prediction image, namely the third image.

[0014] Specifically, the S4 includes: S41: Calculating the error between the first image and the second image using a loss function; S42: Optimizing the harmonized network model through a back propagation algorithm according to the error.

[0015] Specifically, in S1, the foreground object is a combination of one or more of white clouds, airplanes, and obstacles appearing at low altitude.

[0016] The present invention provides a computer device, comprising a memory and a processor coupled to the memory; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the image harmonization method based on low-altitude simulation requirements.

[0017] The present invention provides a computer-readable storage medium, which is used to store a computer program. When the computer program is executed by a computer, it is used to implement the image harmonization method based on low-altitude simulation requirements.

[0018] Compared to existing technologies, the image harmonization method for low-altitude simulation requirements provided by the present invention only requires a small number of real aerial images, preprocessing the first images that meet the requirements, and then training the harmonization network model to generate a third image that is close to a real aerial image. This method does not require the collection of a large number of real aerial images; instead, it only requires the collection of a small number of real aerial images and the training of the harmonization network model to achieve the realism required for the simulated images. When simulating different low-altitude scenes, it is only necessary to generate foreground objects through a simulator or algorithm, and then combine them with a real low-altitude background to synthesize a simulated image or scene that approaches reality.

[0019] The use of the image harmonization method based on low-altitude simulation needs can not only meet the current needs of eVTOL or UAVs in the low-altitude field in the research and development and testing stage, provide a large number of harmonized simulation scenes, and realize real and effective testing; it can also greatly save the time and cost of collecting real aviation data, and there is no need to use simulators that are expensive and difficult to use. The image harmonization method based on low-altitude simulation needs provided in this application only requires the user to provide simple simulation images or scenes to achieve rich, realistic harmonized images or scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive work, among which: Figure 1 This is a flowchart of the image harmonization method based on low-altitude simulation requirements provided by the present invention; Figure 2 It is a structural block diagram of the image harmonization method based on low-altitude simulation requirements provided by the present invention; Figure 3 is a schematic diagram of the first image collected in an embodiment of the image harmonization method based on low-altitude simulation requirements provided by the present invention; Figure 4 yes Figure 1 The flowchart of S2 shown; Figure 5 is a schematic diagram of the background image generated after the first image is processed in one embodiment; Figure 6 is a schematic diagram of the simulated foreground generated after processing the first image in one embodiment; Figure 7 is a schematic diagram of the second image generated after the first image is processed in one embodiment; Figure 8 yes Figure 1 The flowchart of S3 is shown; Figure 9 is a schematic diagram of an apparatus according to an embodiment of the present invention; Figure 10 It is a schematic structural diagram of a computer-readable storage medium in one embodiment of the present invention. DETAILED DESCRIPTION

[0021] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0022] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0023] In addition, the terms "first," "second," and so on, used in this disclosure are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referenced. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this disclosure, "plurality" means at least two, such as two or three, unless otherwise specifically defined.

[0024] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in this field can implement it. When the combination of technical solutions is mutually contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0025] See also Figure 1 and Figure 2 , Figure 1 This is a flowchart of the image harmonization method based on low-altitude simulation requirements provided by the present invention. Figure 2 The image harmonization method based on low-altitude simulation requirements provided by the present invention is a structural block diagram. The image harmonization method based on low-altitude simulation requirements includes the following steps: S1: Acquire an aerial picture, defined as a first image 1, wherein the first image 1 includes a low-altitude background and a relatively prominent foreground object; S2: performing a disharmony process on the first image 1 to generate a second image 3 with disharmony factors and a foreground mask 15; S3: providing a harmonization network model 5, inputting the second image 3 and the foreground mask 15 into the harmonization network model 5, and generating a harmonization prediction image, which is defined as a third image 7; S4: comparing the first image 1 and the third image 7 to obtain an error, feeding the error back to the harmonized network model 5, and correcting the third image 7; S5: Repeat steps S3 and S4 to optimize the harmonized network model 5 until the third image 7 close to the first image 1 is generated.

[0026] S6: Based on the optimized harmonized network model 5 , a simulated aerial image is input, which is defined as a simulated image. The simulated image is processed by the harmonized network model 5 to generate an aerial image close to the real one.

[0027] Please continue to combine Figures 3 to 7 , the image harmonization method based on low-altitude simulation requirements is further described. Figure 3 The first image 1 is obtained by capturing a real aerial photograph. It should be noted that in S1, the foreground object is a combination of one or more of white clouds, an airplane, or a low-altitude obstacle. Accordingly, in this embodiment, the first image 1 includes the low-altitude background and the foreground object, which is an airplane. In other embodiments, however, the foreground object may be an eVTOL or drone of various types and sizes, or an obstacle such as a balloon, a flying bird, or a high-altitude crane.

[0028] In addition, in order to train a more intelligent and comprehensive harmonization network model 5 and ensure that the simulated image or scene that is closer to reality can be generated, in the process of obtaining the aerial pictures, it is necessary to obtain as many of the first images 1 containing various foreground objects as possible and filter out pictures without any foreground objects to avoid the image harmonization method based on low-altitude simulation requirements from processing too much invalid data and reduce training costs.

[0029] It should be noted that there are various methods for obtaining the aerial images. For example, real aerial images can be collected using various existing low-altitude drones, or virtual aerial images can be generated using simulators or algorithms. Since training the harmonized network model 5 primarily provides the first image 1 and the third image 7 for comparison, the aerial images do not need to be completely realistic. Therefore, a combination of methods can be used to obtain the aerial images, reducing training costs.

[0030] In order to build the simulated image closer to the real scene, it is necessary to extract the foreground object so as to convert and match it to build the simulated image. Figure 4 As shown, the S2 further includes the following steps: S21: marking the foreground object in the first image 1; S22: extracting the marked foreground object, performing a de-harmonization process on the foreground object, generating a simulated foreground 13 and a foreground mask 15; S23: extracting the foreground object from the first image 1 and performing restoration to generate a background image 11; S24: Synthesize the simulated foreground 13 and the background image 11 to generate the second image 3.

[0031] In this embodiment, the airplane is marked in the first image 1 and extracted from the first image 1, which is equivalent to splitting the first image 1 into an image of only the foreground and an image of only the background. Since the background image is missing the airplane, it needs to be further repaired to generate the background image 11, as shown in FIG. Figure 5 The background image 11 is processed in a certain way so that the area where the aircraft is located is integrated with the surrounding environment to form a complete image.

[0032] The proposed aircraft-only image needs to be deharmonized to generate a simulated foreground 13 and a foreground mask 15. Figure 6 As shown, except for the aircraft, the surrounding area is blacked out, forming an image of the same size as the first image 1, containing only the foreground object. Image masking is a common technique in image processing, primarily used to identify or restrict specific areas of an image that require processing. In S22, image masking is used to process the foreground object, facilitating the use of deep learning in the harmonized network model 5. By inputting the mask, image data can be denoised.

[0033] Furthermore, applying image mask processing to the foreground object also facilitates the generation of simulated images for subsequent testing. After the harmonized network model 5 is trained, a large number of simulated images need to be provided via a simulator or algorithm. During the generation of these simulated images, the foreground mask 15 can be provided and paired with various backgrounds, or multiple foreground masks 15 can be combined, thereby providing a wider range of test scenarios.

[0034] To train the harmonized network model 5, please refer to Figure 7 The simulated foreground image 13 is synthesized with the background image 11 to generate the second image 3, as shown in FIG. Figure 7As shown. Since the training of the harmonized network model 5 needs to be based on image data. By synthesizing the simulated foreground 13 and the background image 11, the second image 3 for simulation can be preliminarily obtained. The first image 1 and the second image 3 form a pair of harmonious images and disharmonious images, which can promote the learning and training of the harmonized network model 5, so that the simulated image that tends to be realistic can be output in the end. In this embodiment, by providing the first image 1 and performing disharmonious processing on the first image 1, the disharmonious second image 3 for comparison can be obtained. No simulator or algorithm generation is required, the operation is simple, the algorithm requirements are low, and the overall cost of training the harmonized network model 5 is low.

[0035] It should be noted that the synthesis of the simulated foreground 13 and the background image 11 can be achieved by various algorithms, such as Alpha fusion, pyramid fusion or Poisson fusion. In this embodiment, the MPB algorithm is preferred. The main contents of the MPB algorithm are summarized as follows: 1. Input parameters Provide a source image S (the simulated foreground 13 object area to be fused), a target image T (the background image 11) and a binary mask Ω. The binary mask Ω is the fusion area of S in T, and Ω=1 is the area to be fused.

[0036] 2. Gradient field calculation Define the source image gradient: ∇S=( , ); Define the target image gradient: ∇T=( , ); Define the weighted mixed gradient: G = α∇S + (1−α)∇T; where α is the dynamic weight.

[0037] 3. Constructing the Poisson equation Constructing the Poisson equation: ∇ 2 Φ=div(G)inΩ; where Φ is the fused image to be solved, ∇ 2 is the Laplace operator and div is the divergence.

[0038] Define the boundary condition: Φ|= T| (the pixel value at the boundary of the fusion area is equal to the target image).

[0039] 4. Discretization and solution The Laplace operator ∇ 2 Discrete into five-point template: ∇ 2 Φ≈Φ(i+1,j)+Φ(i−1,j)+Φ(i,j+1)+Φ(i,j−1)−4Φ(i,j); The divergence term div(G) is discretized as: div(G)≈G x (i+1,j)−G x (i,j)+G y (i,j+1)−G y (i, j); Poisson's equation can be formulated as a linear system: AΦ=b.

[0040] Where A is a sparse symmetric matrix (one equation for each pixel), b is div(G) and boundary condition filling; use iterative methods (such as conjugate gradient method, multigrid method) or direct methods (for small-scale problems).

[0041] 5. The output is a seamless fusion image R(p). When p is inside Ω, Φ(p) is taken; when p is outside Ω, T(p) is taken.

[0042] The MPB algorithm uses parameter input, gradient field calculation, component Poisson equation, discretization and solution method steps to finally output a seamless fusion image; it can not only meet the needs of generating the second image 3, but more importantly, it helps to use the MPB algorithm to synthesize a large number of the simulated images after the training of the harmonized network model 5 is completed. That is to say, in S6 and / or S24, the MPB algorithm can be used to synthesize the simulated foreground 13 and the background image 11 into the second image 3 or the simulated image. After the simulated image generated in S6 is input into the harmonized network model 5, it can also respond quickly to recognition and process to generate an aerial image that is close to the real one. Moreover, the background image 11 mainly contains a large amount of blue sky. Using the MPB algorithm to fuse the simulated foreground 13 and the background image 11 can effectively improve the fusion efficiency, without the need for overly complex algorithms, and further save model training costs.

[0043] In S1, real aerial images are collected and further screened, classified, and used. The first image 1 containing the foreground object and the pure background image are screened out. The first image 1 is used to optimize the harmonized network model 5. After optimization, the pure background image and the foreground mask 15 are used in a simulator or algorithm to generate multi-scene, multi-category simulated images. After processing by the harmonized network model 5, a real image or a changed aerial scene is generated.

[0044] Through screening and classification, not only can the collected real aerial image data be accurately applied to ensure the accuracy of model training, but the screened and classified first image 1 and the pure background image can also be effectively used to reduce acquisition costs. After the optimization of the harmonized network model 5 is completed, the simulated image with a large number of scenes is generated. Moreover, based on the simulated image of a large number of scenes, it is possible to further generate changing aerial scenes, expanding the test items in the simulation test phase, including static parameter testing and dynamic obstacle avoidance.

[0045] It should be noted that the changing aerial scenes include not only aerial scenes in which the foreground objects move, flip or change direction relative to the background image, but also aerial scenes in which the background lighting, color changes and image style conversion change.

[0046] Please continue to refer to Figure 1 and Figure 8 The harmonized network model 5 includes an encoder 51 and a decoder 53, and the corresponding step S3 further includes: S31: providing the encoder 51, receiving the second image 3 and the foreground mask 15, encoding to generate image features 55; S32 : providing the decoder 53 , receiving the image features 55 , and generating a harmonized prediction image, namely the third image 7 .

[0047] The harmonization network model 5 can realize network construction through the encoder 51 and the decoder 53, and process the first image 1 and the foreground mask 15 to generate the harmonization prediction map.

[0048] The encoder 51 may select a general feature extraction network to extract image features 55 at different feature dimensions. For example, a similar network model such as Transformer, Unet, ShuffleNet, or RRDBnet may be used as the encoder 51. The image features 55 may be, for example, features of the foreground object or background. Different feature dimensions may include texture, brightness, and color features, and extraction at multiple resolutions can comprehensively and accurately capture the detailed and overall information of the second image 3 and the foreground mask 15, thereby improving the realism of the harmonized prediction image.

[0049] The decoder 53 may select a general image generation network, such as Resnet, Transformer combined with downsampling or deconvolution, and perform decoding, making full use of the features obtained by the encoder 51 to restore the harmonized prediction image.

[0050] It should be noted that the specific network construction and algorithms of the encoder 51 and the decoder 53 can be selected based on needs and will not be described in detail here.

[0051] In order to optimize the harmonized network model 5, the error between the first image 1 and the third image 7 is obtained by comparison, and the error is fed back to the harmonized network model 5 to correct the third image 7. The step S4 further includes the following steps: S41: Calculating the error between the first image 1 and the second image 3 using a loss function; S42: Optimizing the harmonized network model 5 according to the error through a back propagation algorithm.

[0052] The loss function, also known as the error function, can be used to measure the difference between the predicted result and the actual result. It evaluates the performance of the harmonized network model 5 by calculating the degree of inconsistency between the predicted value of the first image 1 and the actual value of the third image 7. Of course, the loss function can use the mean square error (MSE) loss function, mean absolute error (MAE) loss function, or Huber loss function to calculate the error, which will not be further explained here.

[0053] The backpropagation algorithm, combined with the loss function, is used to train the harmonized network model 5. It primarily calculates and updates the parameters of the harmonized network model 5 to minimize the loss between the predicted output and the actual target. Within the deep learning framework, the harmonized network model 5 consists of multiple layers, each containing multiple nodes (or neurons). Errors are propagated back through the network layers from the output layer, and the weights of each layer are adjusted based on the propagated error signal, gradually improving the network's ability to learn the input data. Steps S3 and S4 are repeated, allowing the harmonized network model 5 to undergo multiple iterations of training until convergence or a predetermined number of training rounds are reached.

[0054] The error is equivalent to the image feature 55 extracted during encoding by the encoder 51. Based on the feedback of the error, the harmonized network model 5 can be continuously optimized through the loss function and the back-propagation algorithm.

[0055] Of course, in step S5, a certain error threshold can be set relative to the image feature 55. When the error meets the preset threshold range, the training of the harmonized network model 5 can be stopped, and the trained model can be stored. The specific threshold setting is not limited here.

[0056] In addition, during the training process, the image harmonization method based on low-altitude simulation needs also includes repeating S1-S5, and performing multi-level training on the harmonization network model 5 by providing the first image 1 of multiple scenes to ensure that the generation of the simulation image meets the needs.

[0057] Compared to existing technologies, the image harmonization method for low-altitude simulation requirements provided by the present invention only requires a small number of real aerial images. Preprocessing of the first image 1 that meets the requirements is then performed, and based on the training of the harmonization network model 5, the third image 7 can be generated that approximates a real aerial image. This method eliminates the need to collect a large number of real aerial images; instead, it only requires the collection of a small number of real aerial images to train the harmonization network model 5, thereby achieving the required level of realism in the simulated images. When simulating different low-altitude scenes, a simulator or algorithm is used to generate foreground objects, which are then paired with a realistic low-altitude background to synthesize realistic simulated images or scenes.

[0058] The use of the image harmonization method based on low-altitude simulation needs can not only meet the current needs of eVTOL or UAVs in the low-altitude field in the research and development and testing stage, provide a large number of harmonized simulation scenes, and realize real and effective testing; it can also greatly save the time and cost of collecting real aviation data, and there is no need to use simulators that are expensive and difficult to use. The image harmonization method based on low-altitude simulation needs provided in this application only requires the user to provide simple simulation images or scenes to achieve rich, realistic harmonized images or scenes.

[0059] Please continue reading Figure 9 In one embodiment, a computer device 100 is provided, comprising a memory 101 and a processor 103 coupled to the memory 101. The memory 101 is used to store program data, and the processor 103 is used to execute the program data. The program data includes the following steps: S1: Collect a real aerial picture, defined as a first image 1, where the first image 1 includes a low-altitude background and a relatively prominent foreground object; S2: performing a disharmony process on the first image 1 to generate a second image 3 with disharmony factors and a foreground mask 15; S3: providing a harmonization network model 5, inputting the second image 3 and the foreground mask 15 into the harmonization network model 5, and generating a harmonization prediction image, which is defined as a third image 7; S4: comparing the first image 1 and the third image 7 to obtain an error, feeding the error back to the harmonized network model 5, and correcting the third image 7; S5: Repeat steps S3 and S4 to optimize the harmonized network model 5 until the third image 7 close to the first image 1 is generated.

[0060] Please continue reading Figure 9 In one embodiment, a computer-readable storage medium 300 is provided, on which a computer program 301 is stored. When the computer program 301 is executed, the following steps are implemented: S1: Collect a real aerial picture, defined as a first image 1, where the first image 1 includes a low-altitude background and a relatively prominent foreground object; S2: performing a disharmony process on the first image 1 to generate a second image 3 with disharmony factors and a foreground mask 15; S3: providing a harmonization network model 5, inputting the second image 3 and the foreground mask 15 into the harmonization network model 5, and generating a harmonization prediction image, which is defined as a third image 7; S4: comparing the first image 1 and the third image 7 to obtain an error, feeding the error back to the harmonized network model 5, and correcting the third image 7; S5: Repeat steps S3 and S4 to optimize the harmonized network model 5 until the third image 7 close to the first image 1 is generated.

[0061] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0062] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0063] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. An image harmonization method based on low-altitude simulation requirements, characterized in that: The steps include: S1: Acquire an aerial picture, defined as a first image, where the first image includes a low-altitude background and a relatively prominent foreground object; S2: performing a disharmony process on the first image to generate a second image with disharmony factors and a foreground mask; S3: providing a harmonization network model, inputting the second image and the foreground mask into the harmonization network model, and generating a harmonization prediction image, which is defined as a third image; S4: comparing the first image and the third image to obtain an error, feeding the error back to the harmonized network model, and correcting the third image; S5: Repeat steps S3 and S4 to optimize the harmonized network model until the third image close to the first image is generated.

2. The image harmonization method based on low-altitude simulation requirements according to claim 1 is characterized in that: Also includes: S6: Based on the optimized harmonized network model, a simulated aerial image is input, which is defined as a simulated image. The simulated image is processed by the harmonized network model to generate an aerial image close to the real one.

3. The image harmonization method based on low-altitude simulation requirements according to claim 2, characterized in that: Real aerial pictures are collected, and the first image with the foreground object and the pure background image are screened out. The first image is used to optimize the harmonized network model. After the optimization is completed, the pure background image and the foreground mask are used in a simulator or algorithm to generate the simulated image with scene and category changes. After being processed by the harmonized network model, a real image or a changed aerial scene is generated.

4. The image harmonization method based on low-altitude simulation requirements according to claim 3 is characterized in that: Said S2 further comprises: S21: marking the foreground object in the first image; S22: extracting the marked foreground object, performing a de-harmonization process on the foreground object, generating a simulated foreground and a foreground mask; S23: extracting the foreground object from the first image and performing restoration to generate a background image; S24: Synthesize the simulated foreground image and the background image to generate the second image.

5. The image harmonization method based on low-altitude simulation requirements according to claim 4 is characterized in that: In the step S6 and / or the step S24 , the simulated foreground image and the background image are synthesized into the second image or the simulated image using the MPB algorithm.

6. The image harmonization method based on low-altitude simulation requirements according to claim 4, characterized in that: In the S21, the foreground object in the first image is marked. This is performed synchronously when the aerial picture is obtained in the S1. The foreground object is marked in the first image using an algorithm based on the real or simulated IMU and GPS.

7. The image harmonization method based on low-altitude simulation requirements according to claim 1, characterized in that: The harmonized network model includes an encoder and a decoder, and corresponding to step S3, further includes: S31: providing the encoder, receiving the second image and the foreground mask, and encoding to generate image features; S32: Provide the decoder, receive the image features, and generate a harmonized prediction image, namely the third image.

8. The image harmonization method based on low-altitude simulation requirements according to claim 1, characterized in that: Said S4 includes: S41: Calculating the error between the first image and the second image using a loss function; S42: Optimizing the harmonized network model through a back propagation algorithm according to the error.

9. A computer device, characterized in that: The computer device includes a memory and a processor coupled to the memory; The memory is used to store program data, and the processor is used to execute the program data to implement the image harmonization method based on low-altitude simulation requirements as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store a computer program, and when the computer program is executed by a computer, it is used to implement the image harmonization method based on low-altitude simulation requirements as described in any one of claims 1 to 8.

Images