A method and electronic device for synthesizing infrared simulation data of a landscape

Abstract

This invention discloses a method and electronic device for synthesizing infrared simulation data of a landscape. The method includes: acquiring a visible light image of the landscape to be simulated; automatically dividing the visible light image into regions, with each region corresponding to a maximum connected region with the same outer infrared characteristics; labeling the infrared material of each region, wherein the infrared material includes at least one of roads, vegetation, water surfaces, and buildings; and synthesizing an infrared simulation image of the visible light image based on the infrared material of each region. This embodiment fully considers the infrared characteristics of materials in the infrared simulation image synthesis, ensuring the accuracy of the infrared simulation, and improves the overall efficiency of material segmentation and representation through automation.

Description

Technical Field

[0001] The present invention relates to the field of visual imaging technology, and in particular to a method for synthesizing infrared simulation data of a landscape and an electronic device. Background Technology

[0002] In observation and aiming equipment using infrared sensors (such as manned aircraft, unmanned aircraft, manned vehicles, unmanned vehicles, and other mobile equipment equipped with infrared sensors), daily field training is required. However, daily training lacks necessary hypothetical training grounds and combat scenarios. Building realistic hypothetical training grounds and scenarios is costly and inefficient. Therefore, existing technologies often construct virtual models of training facilities and simulate and synthesize field training scenes or training scenarios based on these virtual models to provide diverse training environments, including synthesizing infrared simulation images collected by infrared sensors.

[0003] Current infrared simulation technologies mainly fall into two categories: One type involves post-processing the image based on computer-simulated visible light images. While this method can approximate infrared effects through image processing, it cannot accurately represent the infrared characteristics of object materials and therefore cannot meet the requirements of scenarios demanding high accuracy in infrared simulation.

[0004] Another type involves manually segmenting and labeling objects, then using the segmentation and labeling results to differentiate and render the computer-generated image based on its materials. This method requires manual object segmentation (including edge detection) and material labeling for each frame of the simulation, which is time-consuming and labor-intensive, and cannot be adapted to situations with a large number of objects. Summary of the Invention

[0005] This invention provides a method and electronic device for synthesizing infrared simulation data of a landscape, in order to solve at least one of the above-mentioned problems.

[0006] In a first aspect, embodiments of the present invention provide a method for synthesizing infrared simulation data of a landscape, characterized in that it includes: Acquire visible light images of the landscape to be simulated; The visible light image is automatically divided into regions, and each region corresponds to a maximum connected region with the same outer infrared characteristics; Label the infrared material of each area, where infrared material includes at least one of the following: roads, vegetation, water surface, and buildings; Based on the infrared material of each region, an infrared simulation image of the visible light image is synthesized.

[0007] In a second aspect, embodiments of the present invention provide an electronic device, the electronic device comprising: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the landscape infrared simulation data synthesis method described in any embodiment.

[0008] In summary, the embodiments of the present invention provide a method for synthesizing infrared simulation data of a landscape, which can achieve the following beneficial effects: 1. It realizes the automatic division of regions with the same infrared material, eliminating the need for manual region segmentation, and making it possible to label materials over a large area. It not only considers infrared materials in infrared image simulation, but also improves the efficiency of dividing regions with the same material, thereby improving the efficiency of the entire material labeling and image synthesis process. It can generate infrared simulation data at the city, provincial, or even national level. 2. This implementation subdivides infrared material types based on the terrain types that need attention in battlefield combat scenarios. It automatically identifies and divides large areas with significant differences in infrared imaging within the terrain according to material type, rather than conventionally identifying individual objects. This is particularly suitable for the infrared simulation scenario in this embodiment. To this end, this embodiment provides cue anchor points with the largest possible area in the SAM model, enabling SAM to focus on the overall characteristics within a block while ignoring details. This achieves the division of large areas of the same material, not only utilizing material information to improve the accuracy of simulation in infrared image synthesis but also improving the overall efficiency of region division and material annotation. Attached Figure Description

[0009] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0010] Figure 1 This is a schematic diagram illustrating the principle of infrared analog image synthesis provided in an embodiment of the present invention; Figure 2 This is a flowchart of a method for synthesizing infrared simulation data of a landscape provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of an online annotation method provided by an embodiment of the present invention; Figure 4 This is a schematic diagram of material encoding for each piece of image data provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of a pixel format for image data provided in an embodiment of the present invention; Figure 6This is a schematic diagram of pixel data of a grassland provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of material encoding at different levels of the same image block provided in an embodiment of the present invention; Figure 8 This is a flowchart illustrating an infrared sensor model for a rendering engine provided in an embodiment of the present invention; Figure 9 This is a schematic diagram illustrating the modeling effect of a virtual infrared sensor provided in an embodiment of the present invention; Figure 10 This is a comparison diagram of the effects of visible light images and infrared simulated images provided in an embodiment of the present invention; Figure 11 This is a flowchart of another method for synthesizing infrared simulation data of a landscape provided in an embodiment of the present invention; Figure 12 This is a schematic diagram of the basic structure of a SAM model provided in an embodiment of the present invention; Figure 13 This is a schematic diagram of an improved SAM model structure provided in an embodiment of the present invention; Figure 14 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0012] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0013] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0014] This invention provides a method for synthesizing infrared simulation data of a terrain. To illustrate this method, the synthesis principle of infrared battlefield data is introduced first. Combined with... Figure 1 The infrared characteristics of the battlefield and targets are affected by a variety of complex environmental factors. The synthesis of infrared battlefield data is carried out under specific global environmental conditions, and the influencing factors can be categorized as follows: 1) The influence of nearby objects, vegetation, and background ground; 2) The overall environmental background has a thermal radiation effect on the target; 3) Interference between the sensor and the target object.

[0015] Based on the aforementioned environmental factors, this embodiment constructs a thermal radiation model specifically for synthesizing infrared simulated images (see subsequent formulas (1) and (2)) according to various infrared imaging parameters required in battlefield combat scenarios, and establishes a corresponding infrared material library. Each infrared material in the material library corresponds to a specific set of infrared imaging parameters. In the imaging calculation, only the infrared material of each pixel needs to be determined to determine the corresponding imaging parameters, thereby synthesizing infrared simulated images. For example, Table 1 is an infrared material library constructed in this embodiment, and the codes for each material type are as follows: Table 1

[0016] As can be seen from the table above, the primary infrared material types are first categorized based on the terrain types that need to be considered in battlefield scenarios. Then, based on the differences in infrared imaging, each primary infrared material type is further subdivided into multiple secondary infrared material types. Buildings are further distinguished primarily based on roof materials, while surface water is further distinguished based on different flow rates and depths. Different materials correspond to different thermal radiation parameters (i.e., the outer-infrared imaging parameters used subsequently), for example: Table 2

[0017] Based on the above principles and material library, Figure 2This is a flowchart of a method for synthesizing infrared simulation data of a landscape according to an embodiment of the present invention. The method is executed by a combination of electronic equipment and manual intervention (e.g., S130), such as... Figure 1 As shown, the specific steps include the following: S110. Obtain the visible light image of the landscape to be simulated.

[0018] This embodiment first obtains landscape image data within a certain geographical range based on the operating range of the device to be simulated (such as observation and aiming equipment), which serves as the data source for the entire method.

[0019] Optionally, the landscape image data can be satellite imagery, aerial photographs, or drone photos covering a large area of ​​a province, country, or even more. Each pixel in the image includes visible light data in the R, G, and B channels. This visible light information will then be transmitted to an infrared sensor, and the infrared image captured by the infrared sensor will be simulated.

[0020] S120. The visible light image is automatically divided into regions, and each region corresponds to a maximum connected region with the same outer infrared characteristics.

[0021] The maximum connected region here does not refer to the region with the largest area selected from multiple connected regions, but rather to the maximum range that a connected region can expand outward. Specifically, in this embodiment, each region has the same infrared characteristics, and the infrared characteristics outside the region are different from those inside. Therefore, each region is a maximum region (if it continues to expand, the internal infrared characteristics will no longer be consistent).

[0022] This embodiment, based on the requirements of infrared scenes, uses AI algorithms to automatically divide the largest connected regions with the same or similar infrared characteristics in the visible light image. Optionally, meta's Segment Anything AI (https: / / segment-anything.com / ) region segmentation technology (hereinafter referred to as the SAM model) can be used to quickly identify and segment the surface image data, improving the efficiency of identification and segmentation, and making subsequent large-area infrared scene annotation possible.

[0023] Specifically, the region identification and segmentation in this embodiment targets large, contiguous areas with significant differences in infrared imaging within the landscape, unlike unconventional object identification. For example, this embodiment divides the entire asphalt road into one region and the entire forest into another, rather than identifying individual trees. Optionally, since material labeling of each region will be performed based on a material library later, material type can also be used as the basis for region segmentation. That is, it is desirable to group consecutive pixels with the same material type into the same region so that the same material can be labeled for each region later.

[0024] In one specific implementation, after activating the SAM model, the approximate range of areas with the same material can be manually determined, and a prompt anchor point within that range can be manually selected, such as by clicking a point within the range or by selecting a rectangle within the range. Since this embodiment aims to segment large areas with the same material, it is preferable to select a rectangle that is as large as possible. The prompt anchor point is input into the SAM model, guiding the model to automatically segment areas containing the prompt anchor point and having the same or similar characteristics as it. This method eliminates the need for manual identification of area boundaries; only preliminary location of the area is required, and the SAM model can automatically determine the area boundaries, achieving a certain degree of automation and improving the efficiency of area segmentation.

[0025] S130, Mark the infrared material of each area.

[0026] like Figure 3 As shown, the S120's AI can quickly identify multiple regions, each with the same infrared material by default, although the infrared materials inside and outside the region differ. You can then select a region to individually set its infrared material category, completing the infrared labeling process.

[0027] S140. Based on the infrared material of each region, synthesize the infrared simulation image of the visible light image.

[0028] Visible light imagery data includes a two-dimensional geographic area based on latitude and longitude. The data within this area is organized into blocks. After texture annotation, the texture distribution of each image data block is effectively divided, such as... Figure 4 As shown, 11 represents the texture of the lake, and 21 represents the texture of the grassland. To record this texture information, the texture codes of the annotations can be written into the image data for later retrieval.

[0029] Optionally, the visible light image data can be in PNG format, where each pixel includes four channels: red, green, blue, and alpha channels. Figure 5 Each channel corresponds to 1 byte of data space. Since the transparency channel has no other use in visible light image data, for efficient integration with subsequent processing, this embodiment directly writes the material code into the transparency channel of the visible light image data. This eliminates the need for additional descriptions and dedicated encoding files to record material information. Furthermore, the transparency channel has 1 byte of space, capable of expressing subdivision information from 0 to 255. Therefore, this embodiment controls the total number of material codes to within 255 to meet the current requirements for computer landscape material classification. For example, the grass code is 21, so the pixels within the grass area are represented as follows: Figure 6 As shown.

[0030] Furthermore, in addition to managing visible light imagery data in blocks, it is also managed in layers of varying fineness, with each layer corresponding to raster units of different scales. Each sub-block within each layer has its own material distribution range, and within the same geographical area, the material labeling corresponding to raster units of different scales should be the same. For example... Figure 7 In the area shown, the material labels of the small-scale grids within the upper left quarter block are consistent with the material labels of the larger-scale grids in the previous level. Therefore, the image data can be written back in multiple levels by referring to the material coding, and the infrared material of each region can be written into the transparency channel of each pixel layer within the block.

[0031] After the materials are written, 3D terrain can be synthesized in the rendering engine by combining elevation and landscape data. A sensor model is then set up based on infrared sensors, and a material infrared feature mapping table is established. The lookup table information is used to calculate the temperature of each material under the current environmental conditions, and atmospheric environmental parameters are fused in. Infrared image synthesis is then performed according to real-time requirements. The infrared material sensor model of the rendering engine includes, but is not limited to, mainstream engines based on OpenGL and DirectX graphics APIs such as Unity and Unreal Engine. Figure 8 The entire implementation process of the sensor model is as follows: Users can create new sensor types and input key parameters (spatial resolution and field of view, temperature measurement accuracy and range, thermal sensitivity and response frequency range) to build virtual infrared sensor models in the rendering engine. The relationship between these real sensor models and the images generated by the virtual sensors is as follows: 1) Spatial resolution and field of view correspond to the viewport size and field of view (FOV) of the camera in the 3D engine. 2) Temperature measurement accuracy corresponds to the sampling accuracy of the material marked in the image data. The higher the test accuracy, the more detailed the material marking is required. 3) The temperature measurement range mainly corresponds to the ambient temperature range in the atmospheric parameters and its influence on the absorption coefficient of the material. When the effect of the ambient temperature on the material exceeds the temperature measurement range, the sensor boundary temperature is displayed. 4) The thermal sensitivity setting corresponds to the entire grayscale simulation range; 5) The response frequency range corresponds to the material band in the material label. Materials not within this band range will not be displayed on the sensor.

[0032] Based on the above relationships, this embodiment performs the following operations for each pixel layer in the virtual infrared sensor: S1. Determine the wavelength range and emissivity of the infrared material for each pixel based on the infrared material code in the transparency channel. Optionally, the wavelength range and emissivity corresponding to each pixel in the current layer can be determined by reading the material code in the transparency channel and looking it up in Table 2.

[0033] S2. Based on the wavelength range and emissivity of the infrared material of each pixel, and the area of ​​the pixels at the same level on the screen, determine the initial grayscale of each pixel in the infrared simulated image. Optionally, unlike the irradiance formula of an actual sensor, this embodiment obtains the grayscale of each rasterized pixel through a virtual infrared sensor, and the specific calculation formula is as follows: (1) The corresponding process for this formula Figure 8 The heat absorption formula in [the text]. Where: The value is a grayscale value, and the range is adjusted between 0 and 255. Lens transmittance is set to a constant of 0.8; The size and area of ​​the material mapped onto the screen through space; L is the distance between the material object and the sensor; It refers to the wavelength range in which the material functions; h is Planck's constant (6.626 × 10⁻⁶). 34J\cdotps); c is the speed of light; λ is the wavelength to be integrated; ε(λ) is the corresponding emissivity of radiation intensity in the material table; To adjust the coefficient, the result range is constrained to between 0 and 255 floating-point numbers.

[0034] The result obtained from the table lookup , and Substituting into formula (1), we can obtain the grayscale value of each pixel in the infrared simulation image, corresponding to... Figure 8 The grayscale value of the target material. The modeling effect at this point is as follows: Figure 9 As shown.

[0035] S3. Correct the initial grayscale of each pixel according to the atmospheric parameters to be simulated, and obtain the final grayscale of each pixel in the infrared simulation image.

[0036] The above modeling did not consider the influence of the atmospheric environment. However, in infrared sensor modeling, the atmospheric environment is one of the most critical factors affecting detection accuracy and effective range. It is not a simple "attenuation coefficient," but a complex system involving multiple physical processes: 1) Atmospheric influences are mainly achieved through three methods: absorption, scattering, and self-emission of infrared radiation.

[0037] 2) Atmospheric transmittance, path radiation and path radiation.

[0038] 3) Environmental and meteorological auxiliary parameters, including temperature, pressure, density vertical profiles and surface temperature, surface emissivity, and wind speed (affecting aerosols).

[0039] Therefore, this embodiment determines the final grayscale value DN of each pixel in the infrared analog image using the following formula: DN= *G* * (2) in, This is an adjustment coefficient used to constrain the overall grayscale value between 0 and 255; G is the gain coefficient, used to expand the grayscale range and avoid pure white or pure black images; Atmospheric spectral transmittance (between 0 and 1), specifically for the infrared band. This can be obtained based on a simplified formula for visibility and distance: ≈exp(-α*R) Where R is the distance between the camera and the target (km), and α is the attenuation coefficient (km). - ¹), which is related to visibility and weather, and the specific values ​​are shown in Table 3: Table 3

[0040] Based on the aforementioned atmospheric parameter fusion formula, the grayscale images of each layer are modified and superimposed to finally synthesize an infrared simulated image. For example, the comparison between visible light image data and infrared simulated image data for a labeled three-way cement road feature is shown below. Figure 10 As shown.

[0041] The entire process described above can also be referenced. Figure 11 Understand. Figure 11 In the document, “1. Large-area aerial photograph library” corresponds to S110, “2.1 AI recognition” in “2 online annotation system” corresponds to S120, “2.2 material annotation” corresponds to S130, and “3. material coding” to “7. data synthesis” correspond to S140.

[0042] In summary, this embodiment provides a method for synthesizing infrared simulation data of a landscape, which can achieve the following beneficial effects: 1. By using AI methods, the region with the same infrared material can be automatically divided without the need for manual region segmentation, which makes it possible to label materials over a large area. It not only takes infrared materials into account in infrared image simulation, but also improves the efficiency of dividing regions with the same material, thereby improving the efficiency of the entire material labeling and image synthesis process. It can generate infrared simulation data at the city, provincial and even national levels. 2. This implementation subdivides infrared material types based on the terrain types that need attention in battlefield combat scenarios. It automatically identifies and divides large areas with significant differences in infrared imaging within the terrain according to material type, rather than conventionally identifying individual objects. This is particularly suitable for the infrared simulation scenario in this embodiment. To this end, this embodiment provides cue anchor points with the largest possible area in the SAM model, enabling SAM to focus on the overall characteristics within a block while ignoring details. This achieves the division of large areas of the same material, not only utilizing material information to improve the accuracy of simulation in infrared image synthesis but also improving the overall efficiency of region division and material annotation.

[0043] 3. A method for embedding material properties into images is proposed. Infrared material information is transferred to the material parsing in the image rendering engine through the image's own transparency channel, without the need for additional external information transmission tools, thus improving transmission efficiency.

[0044] 4. An infrared feature mapping table is proposed to match the infrared materials mentioned above. It can quickly query the infrared parameters and input the external parameters (atmospheric parameters) and internal parameters (material infrared parameters) into the calculation in two steps. When calculating the material's performance, the external parameters do not need to be considered, so as to obtain the infrared image simulation results quickly and effectively.

[0045] Furthermore, in the specific implementation of S120 above, the approximate range of the same material region is first manually determined. Then, a cue anchor point within this range is manually selected, guiding the SAM model to automatically segment the region containing the cue anchor point and having the same or similar characteristics as it. While this method is simple to implement and requires no additional algorithm development, it still requires manually selecting a cue anchor point for each same material region. When the area to be simulated is large, this remains time-consuming and labor-intensive. Therefore, in addition to the above specific implementation, this embodiment also provides another optional implementation: a pre-processing algorithm is developed to automatically identify the approximate range of the same material region and automatically generate SAM cue anchor points within this range, enabling the SAM model to automatically segment the same material region based on the characteristics of the cue anchor points. Optionally, this method may include the following steps: Step 1: Perform low-pass filtering on each color channel of the visible light image. As mentioned above, this embodiment aims to divide large areas of the same material, rather than specific object targets. Therefore, this step uses low-pass filtering to filter out detailed information (such as texture details) while preserving the differences between major material categories.

[0046] Step 2: Based on the contribution of each color channel to each infrared material, construct the infrared features of each filtered pixel. These infrared features include the probability that a pixel belongs to each infrared material. The differences in infrared imaging between different materials are essentially differences in infrared emissivity / thermal radiation parameters, etc. While these characteristic parameters cannot be directly obtained in this embodiment, they can be correlated with the brightness and color distribution of the RGB image. Therefore, this step utilizes the contribution of each color channel to each infrared material to approximately represent the infrared features in the RGB image.

[0047] Optionally, we can first construct the contribution vector [W] of each color channel to each infrared material. R W G W B ], where W R W G and W B W represents the contribution weights of the R, G, and B channels to a certain infrared material, respectively. R +W G +W B =1, and each infrared material corresponds to a contribution vector.

[0048] Then, the filtered color vector of the same pixel is multiplied by the contribution vector of each infrared material to obtain the probability that the same pixel belongs to each infrared material. These probabilities together constitute the infrared feature of the same pixel. Optionally, for a given pixel, the color vector [RGB] of that pixel is calculated and multiplied by the contribution vector of each infrared material [W]. R W G W B The dot product of the pixel and each infrared material is used to measure the similarity between the pixel and the current infrared material. Normalizing the dot product of the pixel and each infrared material (dividing each dot product by the sum of the dot products) yields the probability that the pixel belongs to each infrared material. Arranging these probabilities in order gives the infrared feature of the pixel. Performing the above operation on all pixels yields the infrared feature of each pixel. This feature comprehensively characterizes the infrared properties of the pixel by leveraging different infrared materials, providing an alternative way to express infrared features when specific infrared parameters such as infrared emissivity / thermal radiation are unavailable.

[0049] Furthermore, the contribution vector of each infrared material can be determined as follows: Pixel-level annotations are performed on various infrared materials in a small number of visible light image samples, and the R, G, and B values ​​of each pixel under each infrared material are recorded. Then, taking any infrared material as an example: the average R value of all pixels under that material is calculated to obtain the R mean; similarly, the G mean and B mean can also be obtained. The ratio of the R mean to the sum of the three means is then calculated as the W value corresponding to the current infrared material. R Similarly, W can also be determined. G and W B Performing the above operation on all infrared materials yields the contribution vector for each material.

[0050] Optionally, the contribution vectors of each infrared material can also be determined as follows: First, pixel-level annotations are performed on the infrared materials in a small number of visible light image samples, and the contribution vectors of each color channel to each infrared material are initialized; then, the probability of each pixel belonging to each infrared material is determined based on each contribution vector, and the specific calculation method is the same as the dot product solution mentioned above; then, the infrared material to which each pixel belongs is predicted based on each probability, for example, the infrared material with the highest probability is taken as the infrared material to which the pixel belongs; finally, the element values ​​in each contribution vector are trained based on the difference between the prediction results and the annotation results. For example, the cross-entropy loss function and gradient descent method can be used to iteratively update each element value. Since there are few parameters to be trained, this method only requires annotation of a small number of sample images, and is a lightweight implementation method.

[0051] Step 3: Calculate the infrared feature gradient of each pixel and cluster the pixels into N classes based on the gradient magnitude and gradient direction stability. The purpose of this step is to find regions with relatively stable infrared features through clustering, as these regions are more likely to have the same material.

[0052] Optionally, for a given pixel, the gradient value of the infrared feature can be calculated as follows: and gradient direction :

[0053]

[0054] in, and These represent the horizontal and vertical gradient values ​​of the pixel's infrared features, respectively. Furthermore, there are multiple methods for calculating the gradient of a vector along a certain direction; any one of these methods can be chosen. and For example, the cosine similarity of vectors between adjacent pixels in the horizontal / vertical direction can be calculated, where gradient = 1 - cosine similarity. The specific process is existing technology and will not be elaborated here.

[0055] Then, the variance / entropy of the infrared feature gradient direction within the neighborhood of each pixel can be calculated to characterize the stability of the infrared feature gradient direction of each pixel. That is, for each pixel, the variance / entropy of the gradient direction of all pixels in its neighborhood is calculated. The smaller the variance or entropy, the higher the stability of the gradient direction.

[0056] Finally, a vector composed of the gradient magnitude and gradient direction stability of the same pixel is used to cluster all pixels, dividing each pixel into N classes. Optionally, N>2, which must include a class with a low and uniform gradient field. Ideally, the mean and variance of the gradient of this class are close to 0.

[0057] Step 4: Determine the class with the smallest gradient mean as the same material class, and determine at least one connected region formed by pixels of the same material class as at least one core region of the same material. Among multiple clustering categories, there is usually a class with a small gradient mean and a relatively stable gradient direction; this class corresponds to pixels in typical homogeneous regions. However, due to acquisition errors, calculation errors, etc., even if there is no class that simultaneously satisfies both the smallest gradient mean and the most stable gradient direction, there will certainly be a class with the smallest gradient mean, which can be determined as the same material class. These pixels are not necessarily all continuous in spatial location, but rather form one or more connected regions. Identify the connected regions with excessively small areas, and consider each remaining connected region as a core region of the same material (referred to as a core region of the same material).

[0058] In particular, in step three, N is taken as a natural number greater than 2 in order to set more stringent conditions for screening the core regions of the same material. Unlike simple binary classification, this embodiment further ensures the reliability of the screening results by refining the categories to screen the regions with the highest consistency of infrared features.

[0059] Step 5: Automatically generate cue anchor points within each identical material core region to guide the SAM model in dividing the visible light image, obtaining the largest connected region that contains each cue anchor point and has the same infrared characteristics. Optionally, if there is only one identical material core region, the center point of the identical material core region can be taken as the cue anchor point; or the inscribed rectangle (as large as possible) of the identical material core region can be taken as the cue anchor point (preferred); or the mask of the identical material core region can be directly used as the cue anchor point (preferred). The SAM will divide the continuous region containing the cue anchor point and with similar characteristics to the cue anchor point based on the internal characteristics of the cue anchor point. If there are multiple core regions of the same material, since the center anchor points of the SAM model have the same characteristics by default, in order to avoid confusion in the division, the infrared feature mean values ​​of multiple core regions of the same material can be clustered twice to obtain multiple groups of core regions of the same material with similar materials. Then, for each group: generate a prompt anchor point (center point, inscribed rectangle, or mask of the same material core region) in each core region of the same material in the group. Input all the prompt anchor points in the group into the SAM model at once, and the SAM model outputs the regions of similar materials, with each region containing at least one prompt anchor point.

[0060] Furthermore, since the SAM model inherently analyzes the visual features of the input image, and this embodiment aims to segment regions with consistent infrared characteristics, there is a possibility that visual features may not accurately reflect infrared characteristics, leading to inaccurate segmentation. To address this, in another specific embodiment, the basic structure of the SAM model can be improved by introducing the infrared features constructed in the above embodiment into the SAM model to guide region segmentation through infrared characteristics. Specifically, the basic structure of the SAM model is as follows: Figure 12 As shown, the system includes an image encoder, a cue encoder, and a mask decoder. The image encoder performs global feature extraction on the input RGB image to obtain a high-level image feature map (i.e., the visual features in the image), which is used for subsequent segmentation. The cue encoder encodes user-provided cue information (i.e., cue anchors) into cue features that the model can understand. Its dimensions are aligned with the visual feature space. Cue anchors can be point cue (foreground / background points), bounding box cue, or mask cue. The mask decoder takes visual features and cue features as input and uses them to mask regions that have the same or similar visual features as the cue anchor and include that anchor.

[0061] This embodiment improves upon the basic structure to obtain... Figure 13The model structure is shown below. First, the infrared features of each pixel are arranged sequentially according to their pixel positions, resulting in a three-dimensional infrared feature map. This map has a different dimension than the visual features. Through planar scaling and channel transformation, this map can be converted to have the same dimension (i.e., size) as the visual features output by the image encoder in the SAM model. Specifically, the visual features output by the SAM image encoder are W*H*C in size (corresponding to the width, height, and channel directions of the image, respectively). Since the size differs from the original image, each element in the W*H two-dimensional plane of the visual features often corresponds to a specific image patch in the original image. Therefore, the infrared feature map in the W*H two-dimensional plane can be scaled using upsampling, average pooling, etc., so that the number of elements in the W and H directions of the scaled map is the same as that of the visual features. Then, a 1*1 convolution can be used to transform the number of channels C of the infrared feature map to match the number of channels of the visual features, thus achieving dimensional unification between the two features.

[0062] Then, based on the infrared feature gradient within the pixel block corresponding to each element position of the visual feature, the visual features and the transformed infrared features at each element position are weighted and fused. The smaller the infrared feature gradient, the greater the weight of the infrared feature. After the feature dimension transformation is consistent, this step can integrate the infrared features into the visual features. Optionally, to emphasize the importance of infrared features in the region division of this embodiment, feature fusion can be performed primarily using infrared features and secondarily using visual features. Simultaneously, the infrared feature gradient is smallest in the same material core region, therefore the corresponding infrared feature weight is the largest, while the infrared feature weights at other positions are relatively smaller than those in the same material core region. For example, based on the results of the first clustering, different dynamic weight combinations can be assigned to the same material core region and non-same material core regions; alternatively, the infrared feature gradient of each pixel can be used to construct the following dynamic weights for each pixel:

[0063]

[0064] in, and These represent the infrared feature weights and visual feature weights at the same element position in the W*H two-dimensional plane, respectively. This represents the infrared feature gradient of the original image block corresponding to the element's position (the average infrared feature gradient of each pixel within the block can be taken).

[0065] Then the fusion feature at the position of this element This can be expressed as:

[0066] in, This represents the feature vector at the position of this element in the infrared feature map after dimensionality transformation, composed of the values ​​of each channel. This represents the feature vector at that location, composed of the values ​​of each channel.

[0067] Finally, the fused features and cue features are input into the mask decoder of the SAM model. The mask decoder will output the largest connected region that contains each cue anchor point and has the same infrared characteristics within the domain, which in this embodiment is the region with the same infrared characteristics as each core region of the same material. Of course, "same" here is a relative concept; completely identical or very similar can be considered as identical.

[0068] It is worth mentioning that the above fusion method uses infrared features as the primary basis for region segmentation to obtain the largest connected regions with the same infrared characteristics. Meanwhile, the visual features output by the image encoder can suppress noise in infrared features in areas where infrared features are unstable, providing clearer boundary information. The combination of these two features improves the accuracy of segmenting large regions of the same material in this embodiment. Furthermore, this method does not require retraining the parameters of the SAM model. The image encoder in the SAM model only needs to maintain its original visual feature extraction capability, while the mask decoder only needs to extract region masks with similar features to the prompt anchor points from the input features. This is also an advantage of this scheme, improving the accuracy of material region segmentation, annotation, and infrared simulation with minimal computational resources and time cost.

[0069] In summary, this embodiment constructs infrared features for each pixel to initially characterize material properties based on visible light information and the infrared material information to be segmented. It also determines the same-material core region through the infrared feature gradient field, automatically providing cue anchor points for the SAM model. This improves upon the manual positioning of cue anchor points by automatically positioning them, further enhancing the efficiency of region segmentation. To differentiate it from traditional single-target recognition and segmentation, this embodiment blurs texture details through low-pass filtering and performs two-level clustering on the infrared features. The first-level clustering filters pixels with stable infrared feature fields, further removing noise and texture details. The second-level clustering determines the same-material core region with similar materials, providing cue anchor points with consistent infrared characteristics for the SAM model (especially for larger cue regions). Furthermore, cue anchor points can be input in groups based on the second-level clustering, with each group including multiple cue anchor points. Cue anchor points in the same group can be segmented simultaneously, solving the problem of low efficiency in segmenting cue anchor points one by one and preventing segmentation confusion caused by simultaneous input of different types of anchor points, thus balancing segmentation efficiency and accuracy. Furthermore, to address the issue of inconsistencies between infrared characteristics and the visual features inherent in the SAM model, this embodiment improves the basic structure of the SAM model by incorporating infrared features. Based on the reliability of these infrared features, dynamic and different fusion weights are assigned to the infrared and visual features respectively. This further ensures that the SAM model's mask decoder can output regions with the same infrared characteristics as the cue anchor points (rather than just visually similar), further distinguishing it from the traditional single-target recognition method used in SAM models. Each of these methods ensures accurate material information can be utilized in infrared simulated image synthesis, improving the accuracy of simulation and enhancing the overall efficiency of region segmentation and material annotation. This is particularly suitable for high-precision, large-area simulation scenarios.

[0070] It should be noted that all data involved in this application are information and data that have been authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0071] Figure 14 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 14 As shown, the device includes a processor 60, a memory 61, an input device 62, and an output device 63; the number of processors 60 in the device can be one or more. Figure 14 Taking a processor 60 as an example; the processor 60, memory 61, input device 62, and output device 63 in the device can be connected via a bus or other means. Figure 14 Taking the example of a connection between China and Israel via a bus.

[0072] The memory 61, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the landscape infrared simulation data synthesis method in this embodiment of the invention. The processor 60 executes various functional applications and data processing of the device by running the software programs, instructions, and modules stored in the memory 61, thereby realizing the aforementioned landscape infrared simulation data synthesis method.

[0073] The memory 61 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on terminal usage. Furthermore, the memory 61 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory, or other non-volatile solid-state storage device. In some instances, the memory 61 may further include memory remotely located relative to the processor 60, which can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0074] Input device 62 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the device. Output device 63 may include display devices such as a display screen.

[0075] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the landscape infrared simulation data synthesis method of any embodiment.

[0076] The computer storage medium of this invention can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0077] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0078] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof. Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as C or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A method for synthesizing infrared simulation data of a landscape, characterized in that, include: Acquire visible light images of the landscape to be simulated; The visible light image is automatically divided into regions, and each region corresponds to a maximum connected region with the same outer infrared characteristics; Label the infrared material of each area, where infrared material includes at least one of the following: roads, vegetation, water surface, and buildings; Based on the infrared material of each region, an infrared simulation image of the visible light image is synthesized.

2. The method according to claim 1, characterized in that, The automatic region division of the visible light image, ensuring that each region has the same outer-infrared characteristics, includes: In the visible light image, manually select blocks with the same infrared characteristics; Each block is used as a prompting anchor point to guide the SAM model to automatically divide regions with the same infrared characteristics as each block.

3. The method according to claim 1, characterized in that, The process of synthesizing the infrared simulation image of the visible light image based on the infrared material of each region includes: The infrared material codes of each region are written into the transparency channels of pixels at each level within the region, where different levels correspond to different pixel scales; Perform the following operations on each pixel at the same level: S1. Determine the wavelength range and emissivity of the infrared material of each pixel based on the infrared material code in the transparent channel; S2. Determine the initial grayscale of each pixel in the infrared analog image based on the wavelength range and emissivity of the infrared material of each pixel, as well as the area of ​​the pixels of the same level on the screen. S3. Correct the initial grayscale of each pixel according to the atmospheric parameters to be simulated, and obtain the final grayscale of each pixel in the infrared simulation image.

4. The method according to claim 3, characterized in that, S2 include: The initial grayscale of each pixel in the infrared analog image is determined by the following formula. : in, This represents the lens transmittance of the infrared sensor to be simulated. This represents the area of ​​pixels at the same level on the screen, and L represents the distance from the landscape location corresponding to each pixel to the sensor. Let represent the wavelength range of the infrared material of each pixel, h represent Planck's constant, c represent the speed of light, λ represent the wavelength, and ε(λ) represent the emissivity of the infrared material of each pixel. This represents the adjustment coefficient, used to constrain the range of grayscale values.

5. The method according to claim 3, characterized in that, S3 include: The final grayscale DN of each pixel in the infrared analog image is obtained according to the following formula: DN= *G* * in, G represents the adjustment coefficient, used to constrain the range of grayscale values; G represents the gain coefficient, used to expand the grayscale range. This represents the initial grayscale value of each pixel in the infrared analog image. It represents the transmittance of the atmospheric spectrum.

6. The method according to claim 1, characterized in that, The automatic region division of the visible light image, where each region corresponds to a maximum connected region with the same outer-infrared characteristics, includes: Low-pass filtering is applied to each color channel of the visible light image; Based on the contribution of each color channel to each infrared material, infrared features of each pixel after filtering are constructed, wherein the infrared features include the probability that the pixel belongs to each infrared material. Calculate the infrared feature gradient of each pixel, and cluster each pixel into N classes based on the gradient magnitude and gradient direction stability, where N>2; The class with the smallest gradient mean is identified as the same material class, and at least one connected region composed of pixels of the same material class is identified as at least one kernel region of the same material. Within each core region of the same material, prompt anchor points are automatically generated to guide the SAM model to divide the visible light image, thereby obtaining the largest connected region that contains each prompt anchor point and has the same infrared characteristics.

7. The method according to claim 6, characterized in that, The process of clustering pixels into N classes based on gradient magnitude and gradient direction stability includes: Calculate the gradient direction variance / entropy in the neighborhood of each pixel to characterize the gradient direction stability of each pixel; Cluster the vectors formed by the gradient magnitude and gradient direction stability of each pixel to divide each pixel into N classes.

8. The method according to claim 6, characterized in that, The step of constructing the infrared features of each pixel after filtering based on the contribution of each color channel to each infrared material includes: Construct the contribution vector of each color channel to each infrared material; The color vector of the same pixel after filtering is multiplied by the contribution vector of each infrared material to obtain the probability that the same pixel belongs to each infrared material. The probability together constitutes the infrared feature of the same pixel.

9. The method according to claim 6, characterized in that, The guided SAM model segments the visible light image to obtain the largest connected domains that contain each cue anchor point and have the same infrared characteristics within the domain, including: The visible light image is processed by the image encoder in the SAM model to obtain the visual features of the visible light image. By scaling and / or channel transformation, the spectrum composed of infrared features of each pixel is converted into the same dimension as the visual features; Based on the infrared feature gradient within the pixel block corresponding to each element position of the visual feature, the visual features and the converted infrared features at each element position are weighted and fused. The smaller the infrared feature gradient, the greater the weight of the infrared feature. The fused features and cue features are input into the mask decoder in the SAM model. The mask decoder outputs the largest connected domain that contains each cue anchor point and has the same infrared characteristics within the domain.

10. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the landscape infrared simulation data synthesis method according to any one of claims 1-9.

Images