Multispectral image intelligent simulation system based on three-dimensional model

Abstract

The invention relates to the technical field of image simulation, and discloses a multispectral image intelligent simulation system based on a three-dimensional model, and the system comprises a three-dimensional modeling module, a multispectral imaging module, a light source positioning unit, a light ray tracing unit, a light source intensity calculation unit, a rendering sub-module, and an image synthesis sub-module. On the basis that light source intensity calculation is achieved through a simplified model and based on simple parameters, compared with the prior art, through light source positioning, light path tracking and light source intensity calculation, multi-aspect illumination influences are considered, the influences of surface characteristics and different wavelengths on illumination are comprehensively captured, and the light source intensity calculation accuracy is improved. The illumination effect is improved, the accuracy of light source intensity calculation is improved, data support is provided for subsequent multispectral images and image simulation, and the precision of the multispectral images and image simulation is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image simulation, and in particular to a multi-spectral image intelligent simulation system based on a three-dimensional model. Background Art

[0002] Three-dimensional modeling technology refers to the process of creating three-dimensional objects or scenes using computer software. It is widely used in film production, game development, architectural design, virtual reality and other fields. With the development of technology, three-dimensional modeling is not limited to the creation of static models, but also includes the generation of dynamic models and the design of interactive experiences. Modern three-dimensional modeling software provides a wealth of tools and functions, making the details of the model more refined, the texture more realistic, and can truly reproduce the shape, material and light and shadow effects of the object. Multispectral imaging technology is a technology that obtains information reflected or emitted by a target at multiple specific wavelengths, which can provide detailed information about the composition of the material. Compared with traditional monochrome or color imaging, multispectral imaging can distinguish different types of surface cover, assess plant health, etc. by capturing images in different electromagnetic bands. These bands usually include visible light, near infrared, short-wave infrared, etc., and are widely used in agriculture, geological exploration, environmental monitoring, military reconnaissance and other fields. Finally, by training a neural network model, it is possible to automatically learn and extract features from a large amount of multispectral image data to achieve prediction and simulation of unknown scenes. However, the light intensity calculation method in the prior art usually considers some basic parameters, such as the brightness of the light source, the distance, and the normal vector of the surface. However, this method often ignores other important influencing factors, resulting in inaccurate final results. Specifically, existing methods may have the following limitations:

[0003] Considering only direct lighting: Many existing methods only consider direct lighting from light sources, while ignoring the impact of indirect lighting. This means they cannot correctly simulate ambient light and other indirect lighting effects;

[0004] Simplified surface characteristics: Existing methods often assume that the surface is uniformly reflective, that is, the reflectivity is the same in all directions. In reality, different surfaces have different reflective characteristics and textures, which will affect the lighting effect;

[0005] Ignoring wavelength dependence: Most existing methods do not consider how light intensity varies with wavelength. In multispectral imaging, the light intensity in different bands can vary greatly, so this must be taken into account. Summary of the invention

[0006] In order to solve the above technical problems, the present invention provides a multispectral image intelligent simulation system based on a three-dimensional model, comprising:

[0007] 3D modeling module: used to create a 3D model of the object to be simulated;

[0008] A multispectral imaging module, connected to the three-dimensional modeling module, is used to select different bands to image the three-dimensional model;

[0009] Multispectral imaging module, including the following sub-modules:

[0010] The illumination calculation submodule is used to calculate the illumination intensity according to the light source position and light source intensity;

[0011] The illumination calculation submodule includes the following units:

[0012] A light source positioning unit, used to determine the light source position and light source intensity of the light source;

[0013] A ray tracing unit for tracing the path of light from a light source to a surface of a three-dimensional model;

[0014] A light intensity calculation unit, used for calculating light intensity;

[0015] A rendering submodule, used to render multispectral images according to light intensity;

[0016] An image synthesis submodule is used to synthesize images of light sources with different wavelengths into a multispectral image;

[0017] A physical property setting module, connected to the three-dimensional modeling module, is used to define physical properties of the surface of the object to be simulated in the three-dimensional model; the physical properties include diffuse reflectivity and specular reflectivity;

[0018] Simulation module: connected with the multispectral imaging module and the physical property setting module to generate multispectral images;

[0019] Multi-source heterogeneous data acquisition module: connected with the simulation module and the 3D modeling module, used to acquire RGB images and depth images according to the multispectral images, and to acquire 3D model data and environmental data according to the object to be simulated;

[0020] The analysis module is connected to the multi-source heterogeneous data acquisition module and is used to perform three-dimensional reconstruction based on the multi-spectral image, RGB image, depth image, three-dimensional model data, and environmental data to obtain simulation results.

[0021] Furthermore, the determination of the light source position and light source intensity of the light source includes: configuring the light source for the three-dimensional model, and taking a dynamic image for the three-dimensional model configured with the light source, determining the reference image from the dynamic image, and obtaining the pixel coordinates and average brightness value of the highlight area of ​​the reference image, and mapping the pixel coordinates into three-dimensional coordinates; using a straight line detection algorithm to detect the straight line of the reference image, and obtaining the straight line direction vector according to the detected straight line; obtaining the light source direction vector according to the straight line direction vector; constructing an objective function, minimizing the objective function, obtaining the light source position, and obtaining the distance between the light source and the highlight area according to the light source position and the three-dimensional coordinates; and obtaining the light source intensity according to the average brightness value and the distance between the light source and the highlight area.

[0022] Furthermore, the calculation formula of the light source intensity is: ;

[0023] In the formula, I represents the intensity of the light source, L represents the average brightness value of the highlight area, and d represents the distance between the light source and the highlight area.

[0024] Furthermore, the ray tracing unit includes the following subunits:

[0025] A light initialization subunit, used to initialize one or more light paths according to the light source position and the light source direction vector; and determine the incident direction according to the light source direction vector;

[0026] An intersection test subunit, used to calculate the intersection point between the light path and the surface of the object to be simulated;

[0027] The attribute calculation subunit is used to calculate the intersection point attributes, which include at least the surface normal vector and the emission direction of the intersection point; obtain the half-way vector according to the incident direction and the emission direction, and obtain the incident angle according to the incident direction and the surface normal vector of the intersection point.

[0028] Furthermore, the incident light intensity of the light source reaching each point on the surface of the object to be simulated is determined according to the light source intensity, and the light intensity is calculated according to the incident light intensity, the incident direction, the emission direction, and the surface normal vector of the intersection point.

[0029] Furthermore, the calculation formula of the light intensity is: ;

[0030] In the formula, Representative band The light intensity under Representative band The direction vector of the light source is The incident light intensity of the light source reaching the i-th intersection point on the surface of the object to be simulated, represents the bidirectional reflectance distribution function, Representative band From the incident direction The incident light is in the outgoing direction The probability of emission, represents the surface normal vector of the i-th intersection point, Represents the incident direction The solid angle increment caused by the change in the upward direction, represents the incident direction, Represents the emission direction.

[0031] Furthermore, the calculation formula of the incident light intensity is: ;

[0032] Where I represents the light source intensity, stands for transfer factor, Represents the attenuation factor.

[0033] Furthermore, the bidirectional reflectance distribution function is: ;

[0034] In the formula, represents the diffuse reflectance, represents the specular reflection coefficient, Representative band The diffuse reflectance under represents the Fresnel term, represents the microsurface distribution function, represents the geometric occlusion function, represents the angle of incidence, h represents the half-distance vector, represents pi, Representative band The diffuse reflectance under Representative band The specular reflection coefficient under Represents the incident direction The incident light is in the outgoing direction The geometric occlusion function.

[0035] Furthermore, the micro-surface distribution function is: ;

[0036] In the formula, represents the roughness coefficient, Represents the angle between the half-distance vector and the surface normal vector at the intersection point.

[0037] Furthermore, the geometric occlusion function is: ;

[0038] In the formula, Represents the minimum value.

[0039] The embodiments of the present invention have the following technical effects:

[0040] The present invention replaces the existing method of calculating light intensity by simplifying the model and based on simple parameters. The present invention calculates light intensity by locating the light source, tracing the light path, and compared with the existing technology, the present invention takes into account various aspects of light influence, and comprehensively captures the influence of surface characteristics and different wavelengths on light, thereby improving the lighting effect and the accuracy of light intensity calculation, providing data support for subsequent multispectral images and image simulations, thereby greatly improving the accuracy of multispectral images and image simulations.

[0041] By introducing the bidirectional reflectance distribution function, the reflective characteristics of various surfaces can be simulated more accurately, and the geometric information of the intersection points can be considered to more accurately calculate the reflection path of the light on the surface, thereby improving the accuracy of lighting calculations. By considering the wavelength dependence, the present invention can more realistically simulate the reflection effects of different colors of light and improve the realism of lighting. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 It is a structural schematic diagram of a multi-spectral image intelligent simulation system based on a three-dimensional model provided by an embodiment of the present invention;

[0044] Figure 2 A structural block diagram of a multispectral imaging module of a multispectral image intelligent simulation system based on a three-dimensional model provided by an embodiment of the present invention;

[0045] Figure 3 A structural block diagram of an illumination calculation submodule of a multispectral image intelligent simulation system based on a three-dimensional model provided by an embodiment of the present invention;

[0046] Figure 4 A structural block diagram of a ray tracing unit of a multispectral image intelligent simulation system based on a three-dimensional model provided by an embodiment of the present invention;

[0047] Figure 5 It is a structural schematic diagram of an electronic device provided by the implementation of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0049] Figure 1 Schematic diagram of a multi-spectral image intelligent simulation system based on a three-dimensional model provided by an embodiment of the present invention. Figure 1 , specifically including:

[0050] 3D modeling module: used to create a 3D model of the object to be simulated.

[0051] The multispectral imaging module is connected to the three-dimensional modeling module and is used to select different bands to image the three-dimensional model.

[0052] Import the 3D model in ENVI, select the visible light bands (red, green, blue) and near-infrared bands, adjust the imaging parameters such as exposure time and resolution, start the imaging process, and perform multispectral imaging of the simulated object.

[0053] like Figure 2 As shown, the multispectral imaging module includes the following submodules:

[0054] The illumination calculation submodule is used to calculate the illumination intensity according to the light source position and light source intensity.

[0055] like Figure 3 As shown, the illumination calculation submodule includes the following units:

[0056] The light source positioning unit is used to determine the light source position and light source intensity of the light source.

[0057] The light source position and light source intensity of the light source are determined, including: configuring the light source for the three-dimensional model, and shooting a dynamic image for the three-dimensional model configured with the light source, determining the reference image from the dynamic image, and obtaining the pixel coordinates and average brightness value of the highlight area of ​​the reference image, and mapping the pixel coordinates to three-dimensional coordinates; using a straight line detection algorithm to detect the straight line of the reference image, and obtaining the straight line direction vector according to the detected straight line; obtaining the light source direction vector according to the straight line direction vector; constructing an objective function, minimizing the objective function, obtaining the light source position, and obtaining the distance between the light source and the highlight area according to the light source position and the three-dimensional coordinates; and obtaining the light source intensity according to the average brightness value and the distance between the light source and the highlight area.

[0058] In this embodiment, in the process of determining a reference image from a plurality of dynamic images, first, the evaluation index is determined: the symmetry of the highlight area, the shape factor, and the dynamic image contrast. The evaluation index is calculated, and a comprehensive score is calculated based on the evaluation index. The dynamic image of the highlight area with the highest comprehensive score is determined as the reference image. Specifically:

[0059] Symmetry calculation of highlight areas of dynamic images: First, perform brightness analysis on each dynamic image to extract highlight areas, and use threshold segmentation algorithm to convert highlight areas into binary highlight images. Based on the binary highlight images, perform two-dimensional Fourier transform to convert the spatial domain into the frequency domain, and move the zero-frequency component of the spectrum to the center position to better observe the symmetry of the spectrum. Calculate the symmetry of the spectrum in the horizontal and vertical directions, flip the spectrum horizontally and vertically, calculate the difference between the flipped spectrum and the original spectrum, and calculate the symmetry score. The smaller the score, the better the symmetry.

[0060] ;

[0061] In the formula, Indicates frequency The frequency domain value of Represents the pixel value of the binary highlight image in the spatial domain, M and N represent the width and height of the image respectively, j represents the imaginary unit, is the circumference of a circle, .

[0062] ;

[0063] Where S represents the symmetry score, Represents the frequency domain value of the image symmetrically about the horizontal axis, Represents the frequency domain value of the image symmetrically about the vertical axis.

[0064] In this embodiment, the value range of the symmetry score is 1-Q. The first threshold, the second threshold, and the third threshold are constructed according to the value range of the symmetry score, and the symmetry score is determined according to the first threshold, the second threshold, and the third threshold. Further, the first threshold is [1, Q / 4], the second threshold is (Q / 4, 3Q / 4], and the third threshold is (3Q / 4, Q], when the symmetry score is within the first threshold range, the symmetry score is determined to be c, when the symmetry score is within the second threshold range, the symmetry score is determined to be h, and when the symmetry score is within the third threshold range, the symmetry score is determined to be w, c>h>w.

[0065] Fourier transform converts the image from the spatial domain to the frequency domain, and the spectrum reflects the global characteristics of the image. By evaluating the symmetry of the spectrum, we can have a more comprehensive understanding of the overall symmetry of the image, rather than just the local symmetry.

[0066] Calculation of the shape factor of the highlight area: The highlight area that is closer to a circle can be determined as the reference image. Use ellipse fitting to evaluate the shape factor. The closer the ratio of the major axis to the minor axis of the ellipse fitting is to 1, the closer the shape is to a circle.

[0067] ;

[0068] Where L represents the shape factor, represents the length of the major axis of the fitted ellipse, Represents the length of the minor axis of the fitted ellipse.

[0069] This embodiment constructs a shape factor threshold, and determines the shape factor score according to the shape factor threshold. Preferably, the shape factor threshold is set to 0.8, and the shape factor score is determined to be 0.9 for a shape factor greater than or equal to 0.8, and the shape factor score is determined to be 0.3 for a shape factor less than 0.8.

[0070] Dynamic image contrast is calculated by calculating the average brightness difference between the highlight area and the background area of ​​the dynamic image. The greater the difference, the greater the contrast.

[0071] ;

[0072] Where P represents contrast, Represents the average brightness value of the highlight area. Represents the average brightness value of the background area.

[0073] ;

[0074] SC stands for comprehensive score.

[0075] Comprehensive scoring can more accurately select the best reference images and improve the accuracy of subsequent image processing and analysis. Comprehensive scoring is based on objective mathematical calculations, which reduces the subjective factors of human judgment and improves the reliability of the results.

[0076] The dynamic image with the maximum comprehensive score is used as the reference image of this embodiment, and the average brightness value of the highlight area of ​​the reference image is calculated.

[0077] Furthermore, the edge detection algorithm is used to identify the edge of the reference image, and the Hough transform is used to detect the straight line in the image. The Hough transform can convert the points in the image space into curves in the parameter space, thereby finding the straight line. For each detected straight line, its two endpoints (x1, y1) and (x2, y2) can be obtained, and the straight line direction vector is calculated: .

[0078] For each straight line, its normal direction vector can be obtained by rotating it 90 degrees. According to the incident direction, the straight line direction vector can be determined to be or When multiple straight lines are detected, the average value of all straight line normals is calculated as the overall light source direction vector.

[0079] In this embodiment, the preset light source position is (x, y, z), where z represents the height of the light source relative to the reference image plane. The objective function is constructed to minimize the sum of the squares of the distances from each pixel in all highlight areas to the light source. i ,y i , z i ) is the position of the i-th pixel, then the objective function is: ; N represents the maximum number of pixels in the highlight area of ​​the reference image. Use gradient descent, Newton method or other numerical optimization methods to solve the minimum value of the objective function to obtain the light source position.

[0080] The significance of the objective function design is to find the light source position so that the sum of the squares of the distances from each pixel in all highlight areas to the light source position is minimized, that is, the minimum value of the objective function is the light source position.

[0081] The distance between each pixel in the highlighted area of ​​the reference image and the light source is calculated based on the Euclidean distance.

[0082] The calculation formula of the light source intensity is: ;

[0083] In the formula, I represents the intensity of the light source, L represents the average brightness value of the highlight area, and d represents the distance between the light source and the highlight area.

[0084] A ray tracing unit is used to trace the path of light from the light source to the surface of the 3D model.

[0085] like Figure 4 As shown, the ray tracing unit includes the following sub-units:

[0086] The light initialization subunit is used to initialize one or more light paths according to the light source position and the light source direction vector; and determine the incident direction according to the light source direction vector.

[0087] The light source position obtained by the objective function is: w = (x s ,y s , z s ), the light source direction vector r=(r x , r y , r z ) as the starting point of the light path, construct the path equation: R(t)=w+tr, R(t) represents the end point of the path at a distance t along the light direction. This embodiment determines the light path according to the path equation. This embodiment determines the incident direction according to the light source direction vector r.

[0088] The intersection test subunit is used to calculate the intersection point between the light path and the surface of the object to be simulated.

[0089] In practical applications, the surfaces of most 3D models are complex polygonal meshes. These meshes can represent various shapes, such as cars, people, buildings, etc. The advantage of polygonal meshes is that they can flexibly represent arbitrarily complex geometric shapes, and can be decomposed into multiple simple triangles through triangulation, which is convenient for ray tracing and other geometric calculations.

[0090] A polygonal mesh is usually composed of a plurality of triangles, each of which has three vertices. In order to calculate the intersection point between the light ray and the polygonal mesh, this embodiment calculates the intersection point by using the Morley-Klein algorithm.

[0091] The three vertices of the preset polygonal mesh are s0, s1, and s2; the triangle edge vectors are calculated: e1=s1-s0, e2=s2-s0. Specifically, e1 represents the vector from the triangle vertex s0 to the vertex s1, and e2 represents the vector from the triangle vertex s0 to the vertex s2. The auxiliary vector U is calculated: U=r×e2. This formula represents the cross product of the light source direction vector and the triangle edge vector. This auxiliary vector is perpendicular to the light source direction and one side of the triangle. The auxiliary vector T is calculated: T=w-s0. This formula represents the vector from the light source starting point w to the triangle vertex s0. The auxiliary vector q is calculated: q=T×e1. This vector is also perpendicular to the light source direction and one side of the triangle. The determinant is calculated: a=e1×U; a is a scalar used to check whether the light source is parallel to or coincides with the triangle. This embodiment presets a threshold. , ,when , then the light source is parallel to or coincides with the triangle, and there is no intersection. , then there is an intersection point between the light source and the triangle. It is not enough to determine whether the light source intersects the plane where the triangle is located only by the determinant a and the threshold. Even if the light intersects the plane, the intersection point may fall outside the triangle. In this case, the intersection point is not a valid intersection point because the light does not actually intersect the triangle. In this regard, this embodiment further screens the intersection points determined above: calculate the above intersection point (f, g, t) to determine whether the intersection point is inside the triangle: ; ; . Determine whether f is greater than or equal to 0 and less than or equal to 1, and whether g is greater than or equal to 0 and less than or equal to 1. ,at the same time, 0 (representing the intersection point is in the direction of the light source); if (the above conditions must be met at the same time), the intersection point is inside the triangle and is a valid intersection point. If not, the intersection point is outside the triangle and is an invalid intersection point. Finally, , and substitute into the light path equation to get the specific intersection point.

[0092] The attribute calculation subunit is used to calculate the intersection point attributes, which include at least the surface normal vector and the emission direction of the intersection point; obtain the half-way vector according to the incident direction and the emission direction, and obtain the incident angle according to the incident direction and the surface normal vector of the intersection point.

[0093] Surface normal vector at the intersection point It is a vector perpendicular to the surface of the object to be simulated, pointing outward. For a triangular mesh, the surface normal vector of the intersection point is calculated by the cross product of the two edge vectors of the triangle. The outgoing direction is the direction of the light source after reflection or refraction on the surface. The outgoing direction obtained according to the law of reflection for: ;

[0094] Or, the direction of emission obtained according to the law of refraction for: , ;

[0095] and are the refractive indices of the incident and exiting media, respectively.

[0096] Angle of incidence is the angle between the light source direction vector and the surface normal vector at the intersection point: , .

[0097] The half-distance vector is the average direction of the light direction vector and the view direction vector. The view direction is the direction of the line between the position of the camera taking the reference image and the intersection point.

[0098] The incident light intensity of the light source reaching each point on the surface of the object to be simulated is determined according to the light source intensity.

[0099] The calculation formula of the incident light intensity is: ;

[0100] Where I represents the light source intensity, stands for transfer factor, Represents the attenuation factor.

[0101] The light intensity calculation unit is used to calculate the light intensity.

[0102] The illumination intensity is calculated based on the incident illumination intensity, the incident direction, the outgoing direction, and the surface normal vector of the intersection point.

[0103] ;

[0104] In the formula, Representative band The light intensity under Representative band The light source direction vector is The incident light intensity of the light source reaching the i-th intersection point on the surface of the object to be simulated, represents the bidirectional reflectance distribution function, Representative band From the incident direction The incident light is in the outgoing direction The probability of emission, represents the surface normal vector of the i-th intersection point, Represents the incident direction The solid angle increment caused by the change in the upward direction, represents the incident direction, Represents the emission direction.

[0105] In the above formula, It actually refers to the direction of incidence The solid angle increment caused by a small change in refers to the incident direction, and corresponds to a small interval in this direction. When we calculate the integral, we actually accumulate along all possible incident directions. Each accumulation considers the situation within a small range of directions, thus obtaining the total effect within the entire range of incident directions.

[0106] This formula is used to calculate a specific band The core of the light intensity under the condition of light is to consider the influence of all possible incident directions on the final light intensity through integration. The following is a detailed explanation of each component:

[0107] , indicating that in the band Down, from the direction The intensity of light incident on a point on a surface. represents any incident direction, and This quantifies the amount of energy reaching the surface from that direction.

[0108] , called the bidirectional reflectance distribution function, which describes the Reflected to the outgoing direction This function takes into account the physical properties of the material surface, such as diffuse reflectance and specular reflectance, and varies with wavelength, that is, change.

[0109] , which is the incident direction vector and the surface normal vector The dot product of , which reflects the angle of the incident light relative to the surface. When the incident light is perpendicular to the surface, and The angle between them is 0 degrees, at which point their dot product is the largest; as the angle increases, the dot product decreases, which means that the effective energy of the incident light also decreases.

[0110] : refers to the solid angle element corresponding to a small interval in the incident direction. By integrating all possible incident directions, we can fully consider the contribution of light from different directions to the total light intensity.

[0111] This formula emphasizes that by combining the incident light intensity, the bidirectional reflectance distribution function, and the angle of incidence, it is possible to accurately simulate complex lighting environments. This method can not only handle direct lighting, but also take into account the indirect lighting effects after multiple reflections. At the same time, since both the incident light intensity and the bidirectional reflectance distribution function depend on the wavelength, this method is particularly suitable for the simulation of multispectral images because it can capture the differences in lighting characteristics at different wavelengths. In addition, the use of the bidirectional reflectance distribution function makes this method adaptable to a variety of different material properties, making it suitable for a wide range of material types. Ultimately, this physically based rendering method can significantly improve the realism of synthetic images, especially in the performance of complex lighting conditions and fine material details. In short, this formula provides a systematic solution for calculating the light intensity in a specific band, combining the advantages of physical principles and mathematical models, and laying the foundation for high-precision multispectral image simulation.

[0112] ;

[0113] In the formula, represents the diffuse reflectance, represents the specular reflection coefficient, Representative band The diffuse reflectance under represents the Fresnel term, represents the microsurface distribution function, represents the geometric occlusion function, represents the angle of incidence, h represents the half-distance vector, represents pi, Representative band The diffuse reflectance under Representative band The specular reflection coefficient under Represents the incident direction The incident light is in the outgoing direction The geometric occlusion function.

[0114] The formula is built on physical principles and can accurately simulate the optical properties of different materials, including diffuse and specular reflection. By introducing microsurface distribution functions and geometric occlusion functions, the formula can finely capture the influence of surface microstructures, thereby significantly improving the realism of rendering. Specifically, the microsurface distribution function describes the effect of surface microscopic roughness on light reflection, while the geometric occlusion function takes into account the occlusion effect between incident and outgoing light. In addition, the Fresnel term further enhances the accurate simulation of reflectivity changes at different incident angles. These factors work together to enable the formula to not only handle multi-directional reflections in complex lighting environments, but also accurately calculate the reflection probability of light sources in specific directions under different bands 𝜏τ. Therefore, the formula provides a comprehensive and accurate method to calculate the reflection probability of light sources in specific directions, which is of great significance for improving realistic rendering in computer graphics, especially when dealing with high-precision materials and complex lighting conditions.

[0115] ;

[0116] In the formula, represents the roughness coefficient, Represents the angle between the half-distance vector and the surface normal vector at the intersection point.

[0117] Image processing technology is used to analyze the texture features in the reference image and obtain the texture feature values. The texture feature values ​​are averaged and standardized as the roughness coefficient.

[0118] By considering the angle between the half-distance vector and the surface normal vector at the intersection point, the microstructure of the surface of the object to be simulated can be simulated more accurately, resulting in a more realistic visual effect. This helps create material surfaces with rich details and layering. The microsurface distribution function takes into account the degree of microscopic unevenness of the surface, which is particularly important for simulating specular reflections. The roughness coefficient controls the smoothness of the surface, which affects the intensity and range of specular reflections. The microsurface distribution function can produce a more natural transition effect in the highlight area. This is because the microsurface distribution function takes into account the effect of the surface microstructure on light scattering, thus avoiding the hard edges and unnatural highlight shapes that may be produced by traditional models. By treating various types of surfaces in a unified way, whether smooth or rough, consistent results can be obtained. This helps to simplify the material design process and reduce debugging time.

[0119] ;

[0120] In the formula, Represents the minimum value.

[0121] This design enables the geometric occlusion function to effectively reflect the occlusion effect of the light source when it reaches the surface. Specifically, this function determines whether the light is blocked or partially blocked based on the relationship between the incident and outgoing directions and the surface normal of the intersection point. If the dot product result of any direction with the surface normal of the intersection point is less than or equal to zero, that is, the angle formed by the light source and the surface is greater than 90 degrees, then the direction will be considered to be completely blocked, and the corresponding geometric occlusion function value is 0. On the contrary, if both directions are orthogonal or nearly orthogonal to the normal, it means that there is no obvious occlusion phenomenon, and the function value is close to 1. In addition, through the minimum operation, even if there is only a slight occlusion in one direction, the overall occlusion degree will increase, thereby ensuring that the influence of all potential occlusion factors can be correctly reflected in any case. This design method is not only simple and intuitive, but also can largely retain the original advantages while increasing the sensitivity to local environmental information. Therefore, it is very suitable for fast and accurate rendering calculations in complex lighting environments.

[0122] The rendering submodule is used to render the multispectral image according to the light intensity.

[0123] Using the Shader programming capabilities in the physically based rendering engine, a multi-spectral rendering Shader is developed. This Shader can receive different light source intensities as input and adjust the color response of the material according to different wavelengths of light to render a multi-spectral image.

[0124] The image synthesis submodule is used to synthesize images imaged by light sources of different bands into a multispectral image.

[0125] Use Python combined with OpenCV or PIL library to write scripts to read single-channel grayscale images of multiple bands, and then merge them into a multi-channel multispectral image according to specific synthesis rules (such as weighted averaging or physical model-based methods).

[0126] The physical property setting module is connected to the three-dimensional modeling module and is used to define physical properties for the surface of the object to be simulated in the three-dimensional model; the physical properties at least include diffuse reflectivity and specular reflectivity.

[0127] Create a plug-in in the 3D modeling software that allows users to specify different physical properties for various parts of the model, such as diffuse reflectivity, specular reflectivity, etc. These properties can be set directly in the material editor and saved to external files through the export function for use in subsequent simulation modules.

[0128] Simulation module: connected with the multispectral imaging module and the physical property setting module to generate multispectral images.

[0129] Build a high-performance computing framework based on CUDA or OpenCL, which can load the material information provided by the physical property setting module and perform simulation calculations of multispectral images in combination with lighting conditions. During the simulation process, the interaction between light and matter can be simulated according to different lighting angles and intensities, and finally high-quality multispectral images can be output.

[0130] Multi-source heterogeneous data acquisition module: connected with the simulation module and the 3D modeling module, used to acquire RGB images and depth images based on the multispectral images, and to acquire 3D model data and environmental data based on the object to be simulated.

[0131] Develop a set of hardware interface programs that can collect data from various sensors (such as RGB cameras, depth cameras, multispectral cameras, etc.) and transmit these data to the simulation module and 3D modeling module in a unified data format.

[0132] The analysis module is connected to the multi-source heterogeneous data acquisition module and is used to perform three-dimensional reconstruction based on the multi-spectral image, RGB image, depth image, three-dimensional model data, and environmental data to obtain simulation results.

[0133] The acquired multispectral images, RGB images, depth images, and environmental data are analyzed using machine learning algorithms (such as convolutional neural networks (CNNs)) to achieve 3D reconstruction. When training the model, supervised learning can be used to optimize model performance using known 3D scenes as labels. After training, the model can automatically reconstruct a high-precision 3D model based on the input multi-source heterogeneous data.

[0134] Figure 5 Schematic diagram of the structure of an electronic device provided by the present invention. Figure 5 As shown, the electronic device 500 includes one or more processors 501 and a memory 502 .

[0135] The processor 501 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 500 to perform desired functions.

[0136] The memory 502 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 501 may run the program instructions to implement a multi-spectral image intelligent simulation system based on a three-dimensional model and / or other desired functions of any embodiment of the present application described above. Various contents such as initial external parameters, thresholds, etc. may also be stored in the computer-readable storage medium.

[0137] In one example, the electronic device 500 may further include: an input device 503 and an output device 504, which are interconnected via a bus system and / or other forms of connection mechanisms (not shown). The input device 503 may include, for example, a keyboard, a mouse, etc. The output device 504 may output various information to the outside, including early warning prompt information, braking force, etc. The output device 504 may include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, etc.

[0138] Of course, to simplify, Figure 5 Only some of the components related to the present application in the electronic device 500 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 500 may also include any other appropriate components.

[0139] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of a multi-spectral image intelligent simulation system based on a three-dimensional model provided in any embodiment of the present application.

[0140] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0141] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of a multispectral image intelligent simulation system based on a three-dimensional model provided in any embodiment of the present application.

[0142] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0143] It should be noted that the terms used in the present invention are only for describing specific embodiments, rather than limiting the scope of the present application. As shown in the present specification, unless the context clearly indicates an exception, the words "one", "a", "a kind of" and / or "the" do not specifically refer to the singular, but may also include the plural. The terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method or device. In the absence of more restrictions, the elements defined by the sentence "include one..." do not exclude the presence of other identical elements in the process, method or device including the elements.

[0144] It should also be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. Unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", etc. should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be a connection between the two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A multispectral image intelligent simulation system based on a three-dimensional model, characterized in that: Includes the following modules: 3D modeling module: used to create a 3D model of the object to be simulated; A multispectral imaging module, connected to the three-dimensional modeling module, is used to select light sources of different wavelength bands to image the three-dimensional model; Multispectral imaging module, including the following sub-modules: The illumination calculation submodule is used to calculate the illumination intensity according to the light source position and light source intensity; The illumination calculation submodule includes the following units: A light source positioning unit, used to determine the light source position and light source intensity of the light source; A ray tracing unit for tracing the path of light from a light source to a surface of a three-dimensional model; A light intensity calculation unit, used for calculating light intensity; A rendering submodule, used to render multispectral images according to light intensity; An image synthesis submodule is used to synthesize images of light sources with different wavelengths into a multispectral image; A physical property setting module, connected to the three-dimensional modeling module, is used to define physical properties of the surface of the object to be simulated in the three-dimensional model; the physical properties include diffuse reflectivity and specular reflectivity; Simulation module: connected with the multispectral imaging module and the physical property setting module to generate multispectral images; Multi-source heterogeneous data acquisition module: connected with the simulation module and the 3D modeling module, used to acquire RGB images and depth images according to the multispectral images, and to acquire 3D model data and environmental data according to the object to be simulated; The analysis module is connected to the multi-source heterogeneous data acquisition module and is used to perform three-dimensional reconstruction based on the multi-spectral image, RGB image, depth image, three-dimensional model data, and environmental data to obtain simulation results.

2. The multispectral image intelligent simulation system based on a three-dimensional model according to claim 1, characterized in that: The light source position and light source intensity of the light source are determined, including: configuring the light source for the three-dimensional model, and shooting a dynamic image for the three-dimensional model configured with the light source, determining the reference image from the dynamic image, and obtaining the pixel coordinates and average brightness value of the highlight area of ​​the reference image, and mapping the pixel coordinates to three-dimensional coordinates; using a straight line detection algorithm to detect the straight line of the reference image, and obtaining the straight line direction vector according to the detected straight line; obtaining the light source direction vector according to the straight line direction vector; constructing an objective function, minimizing the objective function, obtaining the light source position, and obtaining the distance between the light source and the highlight area according to the light source position and the three-dimensional coordinates; and obtaining the light source intensity according to the average brightness value and the distance between the light source and the highlight area.

3. The multispectral image intelligent simulation system based on a three-dimensional model according to claim 2, characterized in that: The calculation formula of the light source intensity is: ； In the formula, I represents the intensity of the light source, L represents the average brightness value of the highlight area, and d represents the distance between the light source and the highlight area.

4. The multispectral image intelligent simulation system based on a three-dimensional model according to claim 3, characterized in that: The ray tracing unit includes the following sub-units: A light initialization subunit, used to initialize one or more light paths according to the light source position and the light source direction vector; and determine the incident direction according to the light source direction vector; An intersection test subunit, used to calculate the intersection point between the light path and the surface of the object to be simulated; The attribute calculation subunit is used to calculate the intersection point attributes, which include at least the surface normal vector and the emission direction of the intersection point; obtain the half-way vector according to the incident direction and the emission direction, and obtain the incident angle according to the incident direction and the surface normal vector of the intersection point.

5. The multispectral image intelligent simulation system based on a three-dimensional model according to claim 4, characterized in that: Determine the incident light intensity of each point on the surface of the object to be simulated according to the light source intensity; The illumination intensity is calculated based on the incident illumination intensity, the incident direction, the outgoing direction, and the surface normal vector of the intersection point.

6. The multispectral image intelligent simulation system based on a three-dimensional model according to claim 5, characterized in that: The calculation formula of the light intensity is: ； In the formula, Representative band The light intensity under Representative band The direction vector of the light source is The incident light intensity of the light source reaching the i-th intersection point on the surface of the object to be simulated, represents the bidirectional reflectance distribution function, Representative band From the incident direction The incident light is in the outgoing direction The probability of emission, represents the surface normal vector of the i-th intersection point, Represents the incident direction The solid angle increment caused by the change in the upward direction, represents the incident direction, Represents the emission direction.

7. The multi-spectral image intelligent simulation system based on a three-dimensional model according to claim 6, characterized in that: The calculation formula of the incident light intensity is: ； Where I represents the light source intensity, stands for transfer factor, Represents the attenuation factor.

8. The multispectral image intelligent simulation system based on a three-dimensional model according to claim 7, characterized in that: The bidirectional reflectance distribution function is: ； In the formula, represents the diffuse reflectance, represents the specular reflection coefficient, Representative band The diffuse reflectance under represents the Fresnel term, represents the microsurface distribution function, represents the geometric occlusion function, represents the angle of incidence, h represents the half-distance vector, represents pi, Representative band The diffuse reflectance under Representative band The specular reflection coefficient under Represents the incident direction The incident light is in the outgoing direction The geometric occlusion function.

9. The multi-spectral image intelligent simulation system based on a three-dimensional model according to claim 8, characterized in that: The micro-surface distribution function is: ； In the formula, represents the roughness coefficient, Represents the angle between the half-distance vector and the surface normal vector at the intersection point.

10. The multi-spectral image intelligent simulation system based on a three-dimensional model according to claim 8, characterized in that: The geometric occlusion function is: ； In the formula, Represents the minimum value.

Images