Three-dimensional infrared radiation field modeling method based on physical mechanism constraint

Abstract

The invention specifically relates to a three-dimensional infrared radiation field modeling method based on physical mechanism constraint, and the method comprises the steps: obtaining a to-be-processed infrared image, carrying out the recognition of the to-be-processed infrared image to determine a corresponding spatial position parameter and a visual angle parameter, and determining a temperature parameter and an emissivity parameter corresponding to the to-be-processed infrared image; performing coding processing on the spatial position parameter, the visual angle parameter, the temperature parameter and the emissivity parameter to obtain high-dimensional coded data corresponding to each parameter; inputting the high-dimensional coded data into an MLP network, and obtaining an extinction coefficient parameter and a radiation brightness parameter output by the MLP network; wherein the MLP network comprises a first MLP model in a non-activated state and a second MLP model in an activated state; and performing image rendering based on the extinction coefficient parameter and the radiation brightness parameter to obtain a rendered infrared image based on the target visual angle and the target position. According to the method, the problem that the quality of the generated image is poor when NeRF and other technologies are directly applied to the infrared band is solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a three-dimensional infrared radiation field modeling method based on physical mechanism constraints. Background Art

[0002] In related technologies, the three-dimensional neural radiation field generation method based on visible light (Neural Radiance Field, NeRF) can simulate the three-dimensional model of visible light through the network and establish a three-dimensional scattering field that conforms to the law of light field imaging. Visible light imaging focuses on the coupling effect of light intensity of the three RGB channels, which makes NeRF unable to effectively model the key influencing factors of infrared imaging. In addition, when NeRF is applied to infrared-related tasks, the generated images will not only be blurred, but also often accompanied by defects such as ghosting and artifacts, which do not conform to the characteristics of infrared radiation, and even cause overexposure in strong radiation areas, which greatly affects the availability and accuracy of the image.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present invention, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0004] The present invention provides a three-dimensional infrared radiation field modeling method based on physical mechanism constraints, a computer program product, and an electronic device, which can overcome the defects in the prior art to a certain extent.

[0005] Other features and advantages of the present invention will become apparent from the following detailed description, or may be learned in part by practice of the present invention.

[0006] According to a first aspect of the present invention, a three-dimensional infrared radiation field modeling method based on physical mechanism constraints is provided, the method comprising:

[0007] Acquire the infrared image to be processed, identify the infrared image to be processed to determine the corresponding spatial position parameter and viewing angle parameter, and determine the temperature parameter and emissivity parameter corresponding to the infrared image to be processed;

[0008] Encoding the spatial position parameters, viewing angle parameters, temperature parameters, and emissivity parameters to obtain high-dimensional encoded data corresponding to each parameter;

[0009] Input each high-dimensional encoded data into the MLP network to obtain the extinction coefficient parameter and the radiation brightness parameter output by the MLP network; wherein the MLP network includes: a first MLP model in an unactivated state and a second MLP model in an activated state;

[0010] Perform image rendering based on the extinction coefficient parameter and the radiance parameter to obtain a rendered infrared image based on the target viewing angle and target position.

[0011] In some exemplary embodiments, the encoding process for the spatial position parameter, viewing angle parameter, temperature parameter, and emissivity parameter includes:

[0012] Configure the encoding dimensions corresponding to each type of parameter;

[0013] Based on the configured encoding dimensions, perform high-frequency encoding processing on each type of parameter using trigonometric function frequencies to obtain high-dimensional encoded data corresponding to each parameter;

[0014] Construct a multi-dimensional vector matrix based on each high-dimensional encoded data for configuring the multi-dimensional vector matrix as the input parameter of the MLP network.

[0015] In some exemplary embodiments, the inputting the high-dimensional encoded data into the MLP network to obtain the extinction coefficient parameter and radiance parameter output by the MLP network includes:

[0016] Input the high-dimensional encoded data of the spatial position into the first MLP model so that the hidden layer of the first MLP model performs a linear transformation process on the high-dimensional encoded data of the spatial position to obtain the extinction coefficient and the hidden vector output by the first MLP model;

[0017] Input the hidden vector, high-dimensional encoded data of temperature, high-dimensional encoded data of emissivity, and high-dimensional encoded data of viewing angle into the second MLP model so that the hidden layer of the second MLP model performs a non-linear transformation and feature data connection process on each parameter to obtain the radiance parameter output by the second MLP model.

[0018] In some exemplary embodiments, the first MLP model includes an input layer, a hidden layer, and an output layer arranged in sequence; the hidden layer includes two sub-hidden layers with a width of 64 neurons.

[0019] In some exemplary embodiments, the second MLP model includes an input layer, a hidden layer, and an output layer arranged in sequence; the hidden layer includes at least one fully connected layer.

[0020] In some exemplary embodiments, the performing image rendering based on the extinction coefficient parameter and the radiance parameter includes:

[0021] Determine the corresponding transmittance parameter based on the extinction coefficient parameter;

[0022] Define the near boundary and far boundary of the camera ray and construct a camera ray function based on the near boundary and far boundary;

[0023] Construct an infrared volume rendering function based on the camera light function, transmittance parameter, and extinction coefficient; predict the radiance value based on the infrared volume rendering function to complete image rendering.

[0024] In some exemplary embodiments, the method further includes: pre-training an infrared target three-dimensional radiation field model for generating infrared images, including:

[0025] Construct a sample infrared image dataset; wherein, each sample infrared image includes one or more label parameters such as viewing angle, radiance, temperature, material, emissivity.

[0026] Encode the spatial position parameter, viewing angle parameter, temperature parameter, and emissivity parameter of the sample infrared image to obtain high-dimensional encoded data corresponding to each parameter.

[0027] Input each high-dimensional encoded data into an infrared target three-dimensional radiation field model constructed based on an MLP network to obtain the extinction coefficient parameter and radiance parameter output by the model; wherein, the infrared target three-dimensional radiation field model includes: a first MLP model in an unactivated state and a second MLP model in an activated state.

[0028] Perform image rendering based on the extinction coefficient parameter and radiance parameter to obtain a sample rendered infrared image.

[0029] Calculate the radiance loss according to the predicted radiance of the sample rendered infrared image and the true radiance value of the sample infrared image, and perform backpropagation training on the model based on the radiance loss.

[0030] In some exemplary embodiments, the method further includes: the sample infrared image dataset includes at least two different types of basic sample infrared images.

[0031] The method further includes:

[0032] Configure sampling control parameters based on the type of the basic sample infrared image.

[0033] Perform spherical view point interval sampling on the basic infrared sample image within a preset elevation angle range and a preset azimuth angle range according to the sampling control parameters, so as to construct a sample infrared image dataset based on the sampled infrared images.

[0034] Wherein, the sampling control parameters include: the preset elevation angle range is -90° - 90° in the vertical direction; the preset azimuth angle range is -180° - 170° in the horizontal direction; the view point sampling interval is 10°.

[0035] According to a second aspect of the present invention, there is provided a computer program product having a computer program stored thereon, which when executed by a processor implements the above-mentioned three-dimensional infrared radiation field modeling method based on physical mechanism constraints.

[0036] According to a third aspect of the present invention, there is provided an electronic device, comprising:

[0037] a processor; and

[0038] a memory for storing executable instructions of the processor; wherein, the memory is used for storing the executable instructions of the processor; the processor is configured to implement the above-mentioned three-dimensional infrared radiation field modeling method based on physical mechanism constraints when executing the executable instructions.

[0039] According to a fourth aspect of the present invention, there is provided a storage medium having a computer program stored thereon, which when executed by a processor implements the above-mentioned three-dimensional infrared radiation field modeling method based on physical mechanism constraints.

[0040] The three-dimensional infrared radiation field modeling method based on physical mechanism constraints provided by the embodiments of the present invention encodes multiple parameters of an infrared image to obtain corresponding high-dimensional encoded data and uses it as the input data of the model, enabling the input data of the model to have richer and more refined features, which helps the model to more accurately fit data containing high-frequency variations, thereby effectively improving the rendering quality. By setting up a network of two-level MLP models, it effectively simulates the infrared scene mapping, enabling the model to accurately generate images that conform to real infrared radiation characteristics, improving the image generation quality, and solving the problem of poor image generation quality when existing technologies such as NeRF are directly applied to the infrared band.

[0041] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present invention, and are used together with the specification to explain the principles of the present invention. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts.

[0043] Figure 1 A schematic diagram schematically showing a three-dimensional infrared radiation field modeling method according to an exemplary embodiment of the present invention;

[0044] Figure 2 A schematic diagram schematically showing a model architecture according to an exemplary embodiment of the present invention;

[0045] Figure 3 A schematic diagram showing the infrared images corresponding to different means of transportation at different temperatures according to an exemplary embodiment of the present invention;

[0046] Figure 4 A schematic diagram showing the infrared images corresponding to different building walls at different temperatures according to an exemplary embodiment of the present invention;

[0047] Figure 5 A schematic diagram showing a detailed comparison of the vehicle infrared images generated by the NeIRF model and the NeRF model in an exemplary embodiment of the present invention;

[0048] Figure 6 A schematic diagram showing a detailed comparison of the building infrared images generated by the NeIRF model and the NeRF model in an exemplary embodiment of the present invention;

[0049] Figure 7 A schematic diagram showing an electronic device according to an exemplary embodiment of the present invention. Detailed implementation manners

[0050] Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this invention will be more complete and comprehensive, and will fully convey the concept of the example embodiments to those skilled in the art. The features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments.

[0051] In addition, the accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus their repeated description will be omitted. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0052] In related technologies, NeRF simulates visible light three-dimensional models through the network, while the imaging mechanisms of visible light and infrared are significantly different. Infrared imaging is more dependent on information such as the object's temperature, material, and radiation, while visible light imaging focuses on the coupling effect of the light intensity of the three RGB channels. This makes NeRF unable to effectively model the key influencing factors of infrared imaging, so it cannot be directly applied to the infrared band. If it is forcibly applied, the generated image quality will be poor. When NeRF is applied to infrared-related tasks, the generated images will not only be blurred, but also often accompanied by defects such as ghosting and artifacts, which do not conform to the characteristics of infrared radiation, and even cause overexposure in strong radiation areas, greatly affecting the availability and accuracy of the image.

[0053] In view of the shortcomings and deficiencies of the prior art, this example embodiment provides a three-dimensional infrared radiation field modeling method based on physical mechanism constraints, which can solve the problem of poor image quality when the existing technologies such as NeRF are directly applied to the infrared band. Figure 1 As shown, the method may include the following steps:

[0054] Step S11, obtaining an infrared image to be processed, identifying the infrared image to be processed to determine corresponding spatial position parameters and viewing angle parameters, and determining temperature parameters and emissivity parameters corresponding to the infrared image to be processed;

[0055] Step S12, encoding the spatial position parameter, the viewing angle parameter, the temperature parameter, and the emissivity parameter to obtain high-dimensional encoding data corresponding to each parameter;

[0056] Step S13, inputting each high-dimensional encoded data into the MLP network to obtain the extinction coefficient parameter and the radiation brightness parameter output by the MLP network; wherein the MLP network includes: a first MLP model in an unactivated state and a second MLP model in an activated state;

[0057] Step S14, performing image rendering based on the extinction coefficient parameter and the radiation brightness parameter to obtain a rendered infrared image based on a target viewing angle and a target position.

[0058] In the following, each step of the three-dimensional infrared radiation field modeling method based on physical mechanism constraints in this example implementation will be described in more detail with reference to the accompanying drawings and embodiments.

[0059] In step S11, the infrared image to be processed is acquired, and the infrared image to be processed is identified to determine the corresponding spatial position parameters and viewing angle parameters, and the temperature parameters and emissivity parameters corresponding to the infrared image to be processed are determined.

[0060] Exemplarily, a user can create an image processing task for generating an infrared image on a terminal device. A to-be-processed infrared image can be provided as a data basis. First, the COLMAP software can be used to extract the spatial position parameters and viewing angle parameters of the currently input to-be-processed infrared image. Among them, the spatial position parameters can be three-dimensional coordinate information. In addition, the to-be-processed infrared image can also be subjected to image recognition processing to determine the object type in the image, and determine the material information corresponding to the target. Based on this material information, the corresponding emissivity parameter can be determined. In addition, the temperature parameter carried by the infrared image can also be read.

[0061] Specifically, for each parameter, it can be expressed as a multi-dimensional vector-valued function:

[0062]

[0063] Among them, p is the three-dimensional coordinates (x, y, z) after encoding the infrared image; T represents the encoded infrared information (T, ε), T is the Kelvin temperature, and ε is the emissivity of the object, which is related to the material of the object; is the viewing direction (θ, φ), θ is the elevation angle, and φ is the azimuth angle. Lt in the output part is the radiance of the target; α is the extinction coefficient in the Lambert-Beer law.

[0064] Among them, for the 3D position information in the scene, its coordinate values in each direction are obtained in a fine measurement manner and represented by the vector p = (x, y, z). The x, y, and z coordinates here accurately locate the position of each point in the scene in three-dimensional space.

[0065] The viewing direction is represented by two angles (θ, φ), where θ is the elevation angle, which determines the angle of the viewing direction in the vertical plane, while is the azimuth angle, which determines the angle of the viewing direction in the horizontal plane. Different viewing directions will cause significant changes in the visible parts, occlusion relationships, and visual effects of the objects in the scene.

[0066] For the encoded infrared information, the temperature T (Kelvin temperature) and emissivity ε of the object are collected by means of a professional infrared sensor. The temperature T is a key indicator reflecting the thermal state of the object, and different temperature values directly determine the intensity of the infrared radiation emitted by the object. The emissivity ε is obtained through in-depth analysis of the material characteristics of the object. Objects of different materials, such as metals and non-metals, have different emissivities. By encoding and processing this infrared information, key data is provided for the subsequent infrared simulation of the scene.

[0067] In step S12, the spatial position parameters, viewing angle parameters, temperature parameters, and emissivity parameters are encoded to obtain the high-dimensional encoded data corresponding to each parameter.

[0068] Exemplarily, the encoding process for the spatial position parameter, viewing angle parameter, temperature parameter, and emissivity parameter includes:

[0069] Step S121, configure the encoding dimensions corresponding to each type of parameter;

[0070] Step S122, based on the configured encoding dimensions, perform high-frequency encoding processing on each type of parameter using trigonometric function frequencies to obtain high-dimensional encoded data corresponding to each parameter;

[0071] Step S123, construct a multi-dimensional vector matrix based on each high-dimensional encoded data for configuring the multi-dimensional vector matrix as an input parameter of the MLP network.

[0072] Specifically, consider a significant problem: if the original data is directly used as the model input, the model performs poorly in rendering the high-frequency change parts of colors and geometric shapes. In-depth exploration reveals that this is due to the characteristics of the deep network itself, which tends to learn low-frequency functions. Although low-frequency functions can capture the general trends of data, such as the overall contours of objects and broad features like the average distribution of colors, they are significantly insufficient in describing high-frequency details, such as the fine textures on the object surface, the rapid changes of light and shadow at complex geometric edges, and the small structural differences. These high-frequency details are extremely crucial for generating high-fidelity scene images.

[0073] To address the above problems, before inputting each parameter into the model, the encoding dimensions corresponding to each parameter can be configured, and then each parameter can be encoded to obtain multiple different high-dimensional data, which are then combined and organized into a vector matrix. For example, the corresponding encoding dimensions can be configured according to the target type and material in the processed infrared image.

[0074] After comprehensively comparing various existing technologies, this method adopts trigonometric function frequency encoding. This encoding method has many advantages: Firstly, from the perspective of improving the model's expression ability, it uses a combination of sine and cosine functions with a series of different frequencies to generate a large number of features with different periodicities and variation rules; this enables the model to effectively capture complex patterns and subtle features in the input data, solves the problem of low discrimination between adjacent input points in the feature space caused by directly using the original data, and avoids the over-smoothing phenomenon. Secondly, in terms of adaptability, when dealing with spatial data with a large range of variations, such as 3D scene coordinates, by reasonably adjusting the frequency parameters, the encoding can cover the entire data range and capture multi-scale features at the same time, that is, the low-frequency part reflects the overall structure of the data, and the high-frequency part focuses on local details. Thirdly, trigonometric function frequency encoding also has a certain degree of invariance and robustness. In the face of changes in the position and direction of the data, as well as noise interference, the encoded features are relatively stable, can retain key information, ensure the stable performance of the model under different conditions, and facilitate the generalization of the model. Fourthly, considering the computational cost, the trigonometric function calculation process is relatively simple. In the forward and backward propagation stages of the model, the additional overhead of the encoding calculation is small and will not significantly affect the training and inference efficiency of the model. In addition, the principle of trigonometric function frequency encoding is relatively intuitive, easy to understand, and convenient for researchers to analyze the process of the model's processing of different frequency features, providing convenience for the optimization and debugging of the model.

[0075] Specifically, the position coordinates p=(x, y, z) in three-dimensional space are processed using this high-frequency encoding, and the corresponding function is:

[0076] y(p)=(sin(2 0 πp), cos(2 0 πp),..., sin(2 l-1 πp), cos(2 l-1 πp))

[0077] Among them, the configured dimension 1 = 10; at this time, the input is: 2 * 1 dimension * 2 + 3 (original dimension) = 43 dimensions.

[0078] For the viewing direction Using this high-frequency encoding for processing, the corresponding function is:

[0079] y(θ, φ)=(sin(2 0 π(θ, φ)), cos(2 0 π(θ, φ)),..., sin(2 l-1 π(θ, φ)), cos(2 l-1 π(θ, φ)))

[0080] Among them, configuration dimension 1 = 4; at this time, the input is: 2 * l dimension * 2 + 2 (original dimension) = 18 dimensions.

[0081] The temperature and emissivity (T, ε) are processed using this high-frequency encoding, and the corresponding function is:

[0082] y((T, ε)) = (sin(2 0 π(T, ε)), cos(2 0 π(T, ε)),..., sin(2 l-1 π(T, ε)), cos(2 l-1 π(T, ε)))

[0083] Among them, configuration dimension 1 = 4; at this time, the input is: 2 * l dimension * 2 + 2 (original dimension) = 18 dimensions.

[0084] Of course, in other exemplary embodiments of the present invention, other encoding dimension values can also be configured for various parameters.

[0085] By encoding each parameter to obtain high-dimensional data, multiple different high-dimensional data are obtained, and then combined and sorted into a vector matrix. In this way, the input data can show richer and more refined features in the high-dimensional space, enabling the network to more accurately fit data containing high-frequency changes, thereby effectively improving the rendering quality.

[0086] In step S13, the high-dimensional encoded data are input into the MLP network to obtain the extinction coefficient parameter and radiance parameter output by the MLP network; among them, the MLP network includes: a first MLP model in an unactivated state and a second MLP model in an activated state.

[0087] Exemplarily, the inputting the high-dimensional encoded data into the MLP network to obtain the extinction coefficient parameter and radiance parameter output by the MLP network includes:

[0088] Step S131, inputting the high-dimensional encoded data of the spatial position into the first MLP model, so that the hidden layer of the first MLP model performs a linear transformation process on the high-dimensional encoded data of the spatial position to obtain the extinction coefficient and hidden vector output by the first MLP model;

[0089] Step S132, inputting the hidden vector, high-dimensional encoded data of temperature, high-dimensional encoded data of emissivity, and high-dimensional encoded data of viewing angle into the second MLP model, so that the hidden layer of the second MLP model performs a non-linear transformation and feature data connection process on each parameter to obtain the radiance parameter output by the second MLP model.

[0090] Exemplarily, the first MLP model includes an input layer, a hidden layer, and an output layer arranged in sequence; the hidden layer includes two sub-hidden layers each with 64 neurons in width.

[0091] Exemplarily, the second MLP model includes an input layer, a hidden layer, and an output layer arranged in sequence; the hidden layer includes at least one fully connected layer.

[0092] Specifically, each parameter after high-frequency encoding can be input into the MLP network. Among them, as Figure 2 shown, the MLP (Multilayer Perceptron) network can be a neural network model based on Instant-NGP. The MLP network can include a first MLP model and a second MLP model connected in series. Generally speaking, the MLP model includes an input layer, a hidden layer, and an output layer. For the first MLP model, this model is in an unactivated state. The input layer is used to receive high-dimensional encoded data of spatial positions; the hidden layer can include two hidden layers each with 64 neurons in width, and uses a multi-layer neuron structure and connection weights to perform feature extraction and non-linear transformation on the input data, and uses the output layer to output a one-dimensional voxel density and a latent vector.

[0093] For example, the hidden layer can be a linear layer. Neurons in each hidden layer receive the weighted sum of the outputs of the neurons in the previous layer and perform non-linear transformation through an activation function (such as the ReLU function, etc.), so as to learn the complex mapping relationship between the input information and the voxel density and the feature vector. After being processed by two consecutive linear layers, the output layer outputs the voxel density and a latent vector. Among them, the voxel density can be a function only related to the three-dimensional position, which can accurately reflect the material distribution density at different positions in the scene. For example, in a three-dimensional scene containing different materials and objects, it can help determine the scattering and absorption degrees of light when passing through these regions, providing an important physical basis for generating realistic images. Among them, the voxel density is the extinction coefficient. The latent vector is a highly abstract and compressed representation of the input information, containing the key features and patterns of the input data, and can play a role as a bridge for information transmission and integration in the network. For example, the output layer can be a fully connected layer. It can perform a full connection process on the output feature data of the hidden layer, so that the latent vector can contain comprehensive information about the geometric shape, material properties, and lighting conditions of the objects in the scene. These information play an important guiding role in subsequent processes such as predicting radiance, enabling the network to better understand and process the input information, and thus generating more accurate outputs.

[0094] For the second MLP model, which is in an active state, the input data received by the input layer includes: latent vector, high-dimensional temperature encoded data, high-dimensional emissivity encoded data, and high-dimensional view angle encoded data. The hidden layer performs concat (concatenation) processing and non-linear transformation on the data from the input layer through the weighted sum of neurons and activation functions, achieving the fusion of each feature data. For example, the hidden layer may include at least one fully connected layer, and the output layer may be a fully connected layer.

[0095] Specifically, the input of the first MLP model is the sampled position information after high-frequency encoding. First, feature extraction is performed using a linear layer. Each layer of the linear layer contains multiple neurons. The neurons receive the weighted sum of the outputs of the neurons in the previous layer and perform non-linear transformation through an activation function (such as the ReLU function, etc.), thereby learning the complex mapping relationship between the input information and the voxel density and feature vector. After being processed by these 2 linear layers, a one-dimensional voxel density and a hidden variable are output. The network learns the mapping relationship between the input information and the extinction coefficient. The extinction coefficient is a measure of the attenuation degree of light when propagating in a medium and is closely related to factors such as the material and density of the objects in the scene. In a scene containing smoke or semi-transparent objects, the extinction coefficient determines the energy loss of light when passing through these media and is crucial for simulating the light propagation in the subsequent volume rendering process. During the process of the first MLP processing the input information, a latent vector is generated. It can be regarded as a highly abstract and compressed representation of the input information, containing the key features and patterns of the input data. It plays a role as a bridge for information transfer and integration in the network, refines and summarizes the information processed by the first MLP, and then transmits it to the subsequent network layers for further calculation and processing. For example, it may contain comprehensive information about the geometric shape, material properties, and lighting conditions of the objects in the scene. These information play an important guiding role in subsequent processes such as predicting radiance, enabling the network to better understand and process the input information, and thus generating more accurate outputs.

[0096] The second MLP model performs a concat operation on the hidden variables, two-dimensional direction perspective information, and position information obtained from the output of the previous layer. Then, the concatenated information is processed through a fully connected layer. This fully connected layer also performs operations such as weighted sum of neurons and activation functions. The second MLP model further learns the features and mapping relationships related to radiance through the combination of multiple layers of neurons and non-linear transformations using the latent vector. Through the second MLP model, the network can accurately predict the radiance value of an object at a given position and perspective based on the input information (including the spatially encoded position, perspective direction, and the latent vector generated by the first MLP). For example, in a scene with complex lighting, the radiance of an object at different positions and perspectives can vary significantly. The second MLP can capture these complex relationships, thereby providing accurate radiance information for generating realistic images and preparing for subsequent rendering and image generation. Among them, radiance is an important physical quantity describing the intensity of an object's radiant energy and is closely related to factors such as the object's temperature, material, and lighting. Considering that color information is one of the basic elements constituting an image, the output of the second MLP model directly determines the visual effect and realism of the finally generated new perspective image. For example, in a three-dimensional scene with complex lighting, the surface color of an object at different positions and perspectives will change due to various factors such as lighting and material, and the model can capture these complex relationships.

[0097] In step S14, image rendering is performed based on the extinction coefficient parameter and the radiance parameter to obtain a rendered infrared image based on the target perspective and target position.

[0098] Exemplarily, the performing image rendering based on the extinction coefficient parameter and the radiance parameter includes:

[0099] Determine the corresponding transmittance parameter based on the extinction coefficient parameter;

[0100] Define the near boundary and far boundary of the camera ray, and construct a camera ray function based on the near boundary and far boundary;

[0101] Construct an infrared volume rendering function according to the camera ray function, the transmittance parameter, and the extinction coefficient;

[0102] Predict the radiance value based on the infrared volume rendering function to complete the image rendering.

[0103] Specifically, in the rendering stage, based on the extinction coefficient obtained previously, determine the attenuation of the light passing through the scene, calculate the contribution of the light at each sampling point in combination with the radiance, and gradually accumulate to obtain the final image pixel value.

[0104] Based on the Lambert-Beer law, the transmittance τ(d) is a function of the extinction coefficient α and the optical path length d, and the infrared image has a greater correlation with the radiance. The specific formulas are as follows:

[0105] τ(d) = e -αd

[0106] where τ(d) is the transmittance; α is the extinction coefficient; d is the distance that light travels.

[0107] The formula for the camera ray is: r(t) = o + td; define the near boundary t n and the far boundary t f .

[0108] The volume rendering formula follows the camera ray, and the infrared volume rendering formula can be expressed as:

[0109]

[0110] where L t is the radiance; t n is the near boundary, and t f is the far boundary.

[0111] To adapt to the characteristics of discrete data calculation by a computer and combine the particle nature of light, the infrared volume rendering formula is discretized. This process combines the method of stratified sampling, and the formula can be expressed as:

[0112]

[0113] where the distance between t n and t f is divided into N parts; δ i = t i+1 - t i is the distance between adjacent sample points.

[0114] Exemplarily, this method may further include: pre-training an infrared target three-dimensional radiation field model for generating infrared images, including:

[0115] Step S21, constructing a sample infrared image dataset; where each sample infrared image includes one or more label parameters such as viewing angle, radiance, temperature, material, emissivity;

[0116] Step S22, encoding the spatial position parameters, viewing angle parameters, temperature parameters, and emissivity parameters of the sample infrared images to obtain high-dimensional encoded data corresponding to each parameter;

[0117] Step S23: Input each high-dimensional encoded data into the infrared target three-dimensional radiation field model constructed based on the MLP network to obtain the extinction coefficient parameter and the radiance parameter output by the model. Among them, the infrared target three-dimensional radiation field model includes: a first MLP model in an unactivated state and a second MLP model in an activated state.

[0118] Step S24: Perform image rendering based on the extinction coefficient parameter and the radiance parameter to obtain a sample rendered infrared image.

[0119] Step S25: Calculate the radiance loss based on the predicted radiance of the sample rendered infrared image and the true value of the radiance of the sample infrared image, and perform backpropagation training on the model based on the radiance loss.

[0120] Exemplarily, the method further includes: the sample infrared image dataset includes at least two different types of basic sample infrared images. The method further includes:

[0121] Configure sampling control parameters based on the types of the basic sample infrared images.

[0122] Perform spherical view point interval sampling on the basic infrared sample images within a preset elevation angle range and a preset azimuth angle range according to the sampling control parameters, so as to construct a sample infrared image dataset based on the sampled infrared images.

[0123] Among them, the sampling control parameters include: the preset elevation angle range is -90° - 90° in the vertical direction; the preset azimuth angle range is -180° - 170° in the horizontal direction; the view point sampling interval is 10°.

[0124] For example, the infrared target three-dimensional radiation field model (Neural Infrared-RadianceField, NeIRF) can be pre-trained to generate new infrared images according to the specified position and view information. Specifically, the ThermIR dataset can be pre-constructed, and this dataset can include infrared images of at least two types of objects, such as vehicles, buildings, animals, etc. For example, refer to Figure 3 、 Figure 4As shown, the vehicles included in the dataset can be various types of vehicles such as sedans, trucks, SUVs, vans, buses, etc.; the buildings can be various types of buildings such as office buildings, schools, mid-rise residences, high-rise residences, and private houses. For each sample infrared image, the image resolution can be configured to 512*512 pixels. For each target, spherical sampling viewpoints with a vertical range of -90° to 90° (elevation angle) and a horizontal range of -180° to 170° (azimuth angle) are established, with a viewing angle interval of 10°, and a total of 614 viewpoints' images. The training set and test set can be constructed using the sample images, and the training set and test set include images of each target at different viewpoints. In addition, corresponding marker information can be added to the infrared images, including infrared data such as the viewing angle, radiance, temperature, and material (emissivity) of each object. For example, referring to Figure 3 , Figure 4 shown are the infrared images of the vehicle body at different temperatures of 30°C, 40°C, 50°C, 60°C, and 70°C respectively, and the infrared images of the building wall at 15°C, 20°C, 25°C, and 30°C.

[0125] The ThermIR dataset provided by this solution can cover a variety of targets and has the characteristics of multi-viewpoint and multi-factor controllability, providing a data basis for studying the characteristics of infrared targets with multi-physical factor changes, solving the deficiencies of existing datasets in terms of multi-viewpoint and multi-factor controllable image data of infrared radiation, and promoting the development of related research in the field of infrared imaging technology.

[0126] As described in the above embodiment, the infrared target three-dimensional radiation field model can be an MLP network, including a first MLP model in an unactivated state connected in series and a second MLP model in an activated state. Among them, the first MLP model includes an input layer, a hidden layer, and an output layer arranged in sequence; the hidden layer includes two sub-hidden layers with 64 neurons each. The second MLP model includes an input layer, a hidden layer, and an output layer arranged in sequence; the hidden layer includes at least one fully connected layer.

[0127] Among them, the activation function of the model can adopt the ReLU activation function, and the formula can be expressed as:

[0128]

[0129] Specifically, during the iterative training process of the model, the loss function of the model can be the total squared error Loss L based on the rendered pixel radiance and the true pixel radiance, and the formula can be expressed as:

[0130]

[0131] Among them, is the predicted radiance; L0(r) is the true value of the radiance.

[0132] During model training, 290 new perspective test set images can be generated from 324 training images. The training speed can reach 0.2 s / image, and the generation speed can reach 5 s / image. Accordingly, the parameters are set as the learning rate of 5e -4 , the decay factor is set to 500 in 1000 iterations, 64 coarse sampling points and 128 fine sampling points are taken for each radiation line, the number of random rays processed each time is 1024, and the generated background is set to black. NeRF and NeIRF are each trained for 50000 Epochs. The computer environment is the programming language Python, and the framework is implemented by PyTorch, built on Torch2.1.0+Cu121, and implemented on a computer with specific configuration (3.0GHz CPU, 128GB RAM, 1 NVIDIA GeForce RTX 3090 GPU).

[0133] This method sets the ThermIR dataset, compares NeRF and NeIRF qualitatively and quantitatively, and refers to Figure 5 , Figure 6 As shown, it verifies the superiority of NeIRF in generating high-quality infrared images, including better performance in details, radiation characteristics, etc., with the best PSNR and SSIM values under various targets, effectively solving infrared data-related problems. Among them, qualitative evaluation of image quality depends on human eye observation of the visualization results, and quantitative evaluation uses PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity) indicators to evaluate the similarity between the generated image and the source image, so as to comprehensively evaluate the effectiveness and superiority of the NeIRF method.

[0134] The method provided by the present invention determines the key physical factors affecting infrared imaging by combining Planck's law and Lambert-Beer law, uses a neural network to simulate infrared scene mapping, designs a reasonable coding method, network structure, infrared volume rendering formula and loss function, optimizes network parameters, enables the model to accurately generate images that conform to the true infrared radiation characteristics, improves the image generation quality, and solves the problem of poor image generation quality when existing technologies such as NeRF are directly applied to the infrared band. This method can solve the problems of scarce, unopened and difficult-to-share high-value infrared data, generate high-quality multi-factor infrared images that conform to physical laws, realize the rapid and accurate amplification of infrared image samples, provide data support for downstream intelligent tasks such as military reconnaissance and tactical simulation, and overcome the problems of long production cycle of existing infrared target high-fidelity generation methods based on physical simulation and the problem that infrared images generated based on general artificial intelligence do not conform to the physical mechanism of infrared data and are prone to generate distorted images.

[0135] It should be noted that the above-mentioned drawings are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present invention, rather than for limiting purposes. It is easy to understand that the processes shown in the above-mentioned drawings do not indicate or limit the chronological order of these processes. Additionally, it is also easy to understand that these processes can be executed synchronously or asynchronously, for example, in multiple modules.

[0136] Furthermore, in the implementation manner of this example, an electronic device is also provided. The electronic device includes: a processor; and a memory for storing executable instructions of the processor; wherein, the processor is configured to implement the above-mentioned three-dimensional infrared radiation field modeling method based on physical mechanism constraints when executing the executable instructions.

[0137] Figure 7 A schematic diagram of an electronic device suitable for implementing the embodiments of the present invention is shown.

[0138] It should be noted that Figure 4 The shown electronic device 1000 is only an example and should not bring any limitation to the functions and usage scope of the embodiments of the present invention.

[0139] As Figure 7 shown, the electronic device 1000 includes a central processing unit (CPU) 1001, which can execute various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1002 or the program loaded from the storage section 1008 into the random access memory (RAM) 1003. In the RAM 1003, various programs and data required for system operation are also stored. The CPU 1001, ROM 1002, and RAM 1003 are connected to each other through a bus 1004. The input / output (I / O) interface 1005 is also connected to the bus 1004.

[0140] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is mounted on the drive 1010 as needed so that a computer program read from it is installed into the storage section 1008 as needed.

[0141] Specifically, according to an embodiment of the present invention, the processes described below with reference to the flowcharts can be implemented as computer software programs. For example, an embodiment of the present invention includes a computer program product that includes a computer program carried on a storage medium, and the computer program includes program codes for performing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1009, and / or installed from the removable medium 1011. When the computer program is executed by a central processing unit (CPU) 1001, various functions defined in the system of the present application are executed.

[0142] It should be noted that the storage medium shown in the embodiments of the present invention can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, apparatus, or device. In the present invention, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium can also be any storage medium other than a computer-readable storage medium, and this storage medium can send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the storage medium can be transmitted by any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0143] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram can represent a module, a program segment, or a part of code, and the above module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks can occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, and the combination of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0144] The units involved in the embodiments of the present invention can be implemented in software or in hardware, and the described units can also be provided in a processor. Among them, the names of these units do not constitute a limitation to the unit itself in some cases.

[0145] It should be noted that, on the other hand, the present application also provides a storage medium, which can be included in an electronic device; or can exist alone without being assembled into the electronic device. The above storage medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device is caused to implement the methods described in the following embodiments. For example, the described electronic device can implement the steps of the method as shown in Figure 1 、 Figure 2 .

[0146] In one embodiment, the present application provides a computer program product, including a computer program, which when executed by a processor implements the steps in the above method embodiments.

[0147] In addition, the above drawings are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present invention, rather than for limiting purposes. It is easy to understand that the processes shown in the above drawings do not indicate or limit the time sequence of these processes. Additionally, it is also easy to understand that these processes can be executed synchronously or asynchronously in, for example, multiple modules.

[0148] It should be noted that although several modules or units of a device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present invention, the features and functions of the two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0149] Those skilled in the art will readily think of other embodiments of the present invention after considering the specification and practicing the invention herein. The present application aims to cover any variations, uses, or adaptations of the present invention, which follow the general principles of the present invention and include common general knowledge or conventional technical means in the technical field not disclosed in the present invention. The specification and embodiments are only regarded as exemplary, and the true scope and spirit of the present invention are pointed out by the claims.

[0150] It should be understood that the present invention is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present invention is only limited by the appended claims.

Claims

1. A three-dimensional infrared radiation field modeling method based on physical mechanism constraints, characterized in that The method includes: Obtain an infrared image to be processed, identify the infrared image to be processed to determine the corresponding spatial position parameters and viewing angle parameters, and determine the temperature parameter and emissivity parameter corresponding to the infrared image to be processed; Perform encoding processing on the spatial position parameters, viewing angle parameters, temperature parameters, and emissivity parameters to obtain high-dimensional encoded data corresponding to each parameter; Input each high-dimensional encoded data into an MLP network to obtain the extinction coefficient parameter and radiance parameter output by the MLP network; wherein, the MLP network includes: a first MLP model in an unactivated state and a second MLP model in an activated state; Perform image rendering based on the extinction coefficient parameter and radiance parameter to obtain a rendered infrared image based on the target viewing angle and target position.

2. The method according to claim 1, characterized in that, The performing encoding processing on the spatial position parameters, viewing angle parameters, temperature parameters, and emissivity parameters includes: Configure the encoding dimensions corresponding to each type of parameter; Based on the configured encoding dimensions, perform high-frequency encoding processing on each type of parameter using trigonometric function frequencies to obtain high-dimensional encoded data corresponding to each parameter; Construct a multi-dimensional vector matrix based on each high-dimensional encoded data for configuring the multi-dimensional vector matrix as the input parameter of the MLP network.

3. The method according to claim 1 or 2, characterized in that, The inputting the high-dimensional encoded data into the MLP network to obtain the extinction coefficient parameter and radiance parameter output by the MLP network includes: Input the high-dimensional encoded data of the spatial position into the first MLP model, so that the hidden layer of the first MLP model performs linear transformation processing on the high-dimensional encoded data of the spatial position to obtain the extinction coefficient and hidden vector output by the first MLP model; Input the hidden vector, high-dimensional encoded data of the temperature, high-dimensional encoded data of the emissivity, and high-dimensional encoded data of the viewing angle into the second MLP model, so that the hidden layer of the second MLP model performs non-linear transformation and feature data connection processing on each parameter to obtain the radiance parameter output by the second MLP model.

4. The method according to claim 1, wherein The first MLP model includes an input layer, a hidden layer, and an output layer arranged in sequence; the hidden layer includes two sub-hidden layers with 64 neurons in width.

5. The method according to claim 1, wherein The second MLP model includes an input layer, a hidden layer, and an output layer arranged in sequence; the hidden layer includes at least one fully connected layer.

6. The method according to claim 1, characterized in that, The performing image rendering based on the extinction coefficient parameter and radiance parameter includes: Determine the corresponding transmittance parameter based on the extinction coefficient parameter; Define the near boundary and far boundary of the camera ray, and construct a camera ray function based on the near boundary and far boundary; Construct an infrared volume rendering function according to the camera ray function, transmittance parameter, and extinction coefficient; Predict the radiance value based on the infrared volume rendering function to complete the image rendering.

7. The method according to any one of claims 1-6, characterized in that The method further includes: pre-training an infrared target three-dimensional radiation field model for generating infrared images, including: Construct a sample infrared image data set; wherein, each sample infrared image includes one or more label parameters such as viewing angle, radiance, temperature, material, and emissivity; Perform encoding processing on the spatial position parameters, viewing angle parameters, temperature parameters, and emissivity parameters of the sample infrared images to obtain high-dimensional encoded data corresponding to each parameter; Input each high-dimensional encoded data into the infrared target three-dimensional radiation field model constructed based on the MLP network to obtain the extinction coefficient parameter and radiation brightness parameter output by the model; wherein, the infrared target three-dimensional radiation field model includes: a first MLP model in an unactivated state and a second MLP model in an activated state; Perform image rendering based on the extinction coefficient parameter and radiation brightness parameter to obtain a sample rendered infrared image; Calculate the radiation brightness loss based on the predicted radiation brightness of the sample rendered infrared image and the true value of the radiation brightness of the sample infrared image, and perform backpropagation training on the model based on the radiation brightness loss.

8. The method according to claim 7, wherein The method further includes: the sample infrared image dataset includes at least two different types of basic sample infrared images; The method further includes: Configure sampling control parameters based on the types of the basic sample infrared images; Perform spherical viewpoint interval sampling on the basic infrared sample images within a preset elevation angle range and a preset azimuth angle range according to the sampling control parameters, so as to construct a sample infrared image dataset based on the sampled infrared images; Wherein, the sampling control parameters include: the preset elevation angle range is -90° - 90° in the vertical direction; the preset azimuth angle range is -180° - 170° in the horizontal direction; the viewpoint sampling interval is 10°.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the three-dimensional infrared radiation field modeling method based on physical mechanism constraints described in any one of claims 1 to 8.

10. An electronic device, characterized in that, Including: A processor; And A memory for storing executable instructions of the processor; Wherein, the processor is configured to execute the three-dimensional infrared radiation field modeling method based on physical mechanism constraints described in any one of claims 1 to 8 by executing the executable instructions.

Images