A multi-spectral image intelligent simulation system based on a 3D model

Abstract

The present invention relates to the technical field of image simulation, and discloses a multi-spectral image intelligent simulation system based on a three-dimensional model. The system includes: a three-dimensional modeling module, a multi-spectral imaging module, a light source positioning unit, a ray tracing unit, a light source intensity calculation unit, a rendering sub-module, and an image synthesis sub-module. On the basis of the existing technology that simplifies the model and calculates the light source intensity based on simple parameters, the present invention calculates the light source intensity by positioning the light source, tracing the light path, and considering various aspects of the illumination influence compared with the prior art. The present invention comprehensively captures the surface characteristics and the influence of different wavelengths on the illumination, improves the illumination effect and the accuracy of the light source intensity calculation, provides data support for subsequent multi-spectral images and image simulation, and greatly improves the accuracy of multi-spectral images and image simulation.

Description

Technical Field

[0001] The present invention relates to the technical field of image simulation, and particularly to an intelligent multispectral image simulation system based on a three-dimensional model. Background Art

[0002] Three-dimensional modeling technology refers to the process of using computer software to create three-dimensional objects or scenes, which is widely used in fields such as film production, game development, architectural design, virtual reality, etc. With the development of technology, three-dimensional modeling is not limited to the creation of static models, but also includes the generation of dynamic models and the design of interactive experiences. Modern three-dimensional modeling software provides rich tools and functions, making the details of the models more refined, the textures more realistic, and capable of truly reproducing the shape, material, and lighting effects of objects. Multispectral imaging technology is a technology that obtains the information reflected or emitted by a target at multiple specific wavelengths, and can provide detailed information about the substance composition. Compared with traditional monochromatic or color imaging, multispectral imaging can distinguish different types of surface coverings, evaluate the health status of plants, etc. by capturing images in different electromagnetic wave bands. These bands usually include visible light, near-infrared, short-wave infrared, etc., and are widely used in fields such as agriculture, geological exploration, environmental monitoring, and military reconnaissance. Finally, by training a neural network model, features can be automatically learned and extracted from a large amount of multispectral image data to achieve the prediction and simulation of unknown scenes. However, the existing methods for calculating light intensity usually consider some basic parameters, such as the brightness, distance of the light source, and the normal vector of the surface, etc. However, this method often ignores other important influencing factors, resulting in inaccurate final results. Specifically, the existing methods may have the following limitations:

[0003] Only considering direct illumination: Many existing methods only consider the direct illumination from the light source and ignore the influence of indirect illumination. This means that they cannot correctly simulate ambient light and other indirect lighting effects;

[0004] Simplifying surface characteristics: Existing methods often assume uniform reflection on the surface, that is, the reflectivity is the same in all directions. In fact, different surfaces have different reflection characteristics and textures, which will affect the lighting effect;

[0005] Ignoring wavelength dependence: Most existing methods do not consider the variation of light intensity with wavelength. In multispectral imaging, the light intensity in different bands will vary greatly, so this must be taken into account. Summary of the Invention

[0006] In order to solve the above technical problems, the present invention provides an intelligent multispectral image simulation system based on a three-dimensional model, including:

[0007] A three-dimensional modeling module: used to create a three-dimensional model of the object to be simulated;

[0008] A multispectral imaging module, connected to the 3D modeling module, for selecting different bands and imaging the 3D model;

[0009] The multispectral imaging module includes the following sub-modules:

[0010] A lighting calculation sub-module, for calculating the lighting intensity according to the light source position and light source intensity;

[0011] The lighting calculation sub-module includes the following units:

[0012] A light source positioning unit, for determining the light source position and light source intensity of the light source;

[0013] A ray tracing unit, for tracing the light path from the light source to the surface of the 3D model;

[0014] A lighting intensity calculation unit, for calculating the lighting intensity;

[0015] A rendering sub-module, for rendering the multispectral image according to the lighting intensity;

[0016] An image synthesis sub-module, for synthesizing the images of different band light sources into a single multispectral image;

[0017] A physical property setting module, connected to the 3D modeling module, for defining the physical properties of the surface of the object to be simulated in the 3D model; the physical properties include the diffuse reflectivity and specular reflectivity;

[0018] A simulation module: connected to the multispectral imaging module and the physical property setting module, for generating the multispectral image;

[0019] A multi-source heterogeneous data acquisition module: connected to the simulation module and the 3D modeling module, for obtaining the RGB image and depth image according to the multispectral image, and obtaining the 3D model data and environmental data according to the object to be simulated;

[0020] An analysis module, connected to the multi-source heterogeneous data acquisition module, for performing 3D reconstruction according to the multispectral image, RGB image, depth image, 3D model data, and environmental data to obtain the simulation result.

[0021] Further, the determination of the light source position and light source intensity of the light source includes: configuring a light source for the three-dimensional model, taking a dynamic image of the three-dimensional model with the configured light source, determining a reference image from the dynamic image, obtaining the pixel coordinates and average brightness value of the highlighted area of the reference image, and mapping the pixel coordinates to three-dimensional coordinates; using a line detection algorithm to detect the lines of the reference image, obtaining the line direction vectors according to the detected lines; obtaining the light source direction vector according to the line direction vectors; constructing an objective function, minimizing the objective function to obtain the light source position, and obtaining the distance between the light source and the highlighted area according to the light source position and the three-dimensional coordinates; obtaining the light source intensity according to the average brightness value and the distance between the light source and the highlighted area.

[0022] Further, the calculation formula of the light source intensity is:

[0023] ;

[0024] In the formula, I represents the light source intensity, L represents the average brightness value of the highlighted area, and d represents the distance between the light source and the highlighted area.

[0025] Further, the ray tracing unit includes the following sub-units:

[0026] A ray initialization sub-unit, configured to initialize one or more ray paths according to the light source position and the light source direction vector; and determine the incident direction according to the light source direction vector;

[0027] An intersection test sub-unit, configured to calculate the intersection points of the ray paths and the surface of the object to be simulated;

[0028] An attribute calculation sub-unit, configured to calculate the intersection point attributes, where the intersection point attributes at least include the surface normal vector and the outgoing direction of the intersection point; obtain the half-way vector according to the incident direction and the outgoing direction, and obtain the incident angle according to the incident direction and the surface normal vector of the intersection point.

[0029] Further, determine the incident illumination intensity of the light source reaching each point on the surface of the object to be simulated according to the light source intensity, where the illumination intensity is calculated according to the incident illumination intensity, the incident direction, the outgoing direction, and the surface normal vector of the intersection point.

[0030] Further, the calculation formula of the illumination intensity is:

[0031] ;

[0032] In the formula, represents the illumination intensity in the band, represents the incident illumination intensity of the light source with the light source direction vector in the band reaching the i-th intersection point on the surface of the object to be simulated, represents the bidirectional reflectance distribution function, represents the wavelength band at which the probability of light incident from the incident direction being emitted in the emission direction is calculated. represents the surface normal vector at the i-th intersection point, represents the solid angle increment caused by the change in the direction in the incident direction. represents the incident direction, represents the emission direction.

[0033] Furthermore, the calculation formula for the incident light intensity is:

[0034] ;

[0035] wherein, I represents the light source intensity, represents the transmission factor, represents the attenuation factor.

[0036] Furthermore, the bidirectional reflectance distribution function is:

[0037] ;

[0038] wherein, represents the diffuse reflection coefficient, represents the specular reflection coefficient, represents the wavelength band at which the diffuse reflectance is calculated, represents the Fresnel term, represents the micro-surface distribution function, represents the geometric shadowing function, represents the angle of incidence, h represents the half-way vector, represents pi, represents the wavelength band at which the diffuse reflection coefficient is calculated, represents the wavelength band at which the specular reflection coefficient is calculated, represents the geometric shadowing function of the light incident from the incident direction being emitted in the emission direction .

[0039] Furthermore, the micro-surface distribution function is:

[0040] ;

[0041] wherein, represents the roughness coefficient, represents the angle between the half-way vector and the surface normal vector at the intersection point.

[0042] Further, the geometric occlusion function is as follows:

[0043] ;

[0044] In the formula, represents taking the minimum value.

[0045] The embodiments of the present invention have the following technical effects:

[0046] The present invention replaces the existing method of calculating the light intensity by simplifying the model and based on simple parameters. The present invention calculates the light intensity through light source positioning, light path tracing. Compared with the prior art, the present invention considers various aspects of light effects, comprehensively captures the surface characteristics and the influence of different wavelengths on light, improves the light effect and the accuracy of light intensity calculation, provides data support for subsequent multi-spectral images and image simulation, and greatly improves the accuracy of multi-spectral images and image simulation.

[0047] By introducing the bidirectional reflectance distribution function, the reflection characteristics of various surfaces are more accurately simulated, the geometric information of the intersection point is considered, and the reflection path of light on the surface is more accurately calculated, thereby improving the accuracy of light calculation. By considering the wavelength dependence, the present invention can more realistically simulate the reflection effects of different color lights and improve the realism of light. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0049] Figure 1 is a schematic structural diagram of a multi-spectral image intelligent simulation system based on a three-dimensional model provided by an embodiment of the present invention;

[0050] Figure 2 A structural block diagram of a multi-spectral imaging module of a multi-spectral image intelligent simulation system based on a three-dimensional model provided by an embodiment of the present invention;

[0051] Figure 3 A structural block diagram of a light calculation sub-module of a multi-spectral image intelligent simulation system based on a three-dimensional model provided by an embodiment of the present invention;

[0052] Figure 4 A structural block diagram of a ray tracing unit of a multi-spectral image intelligent simulation system based on a three-dimensional model provided by an embodiment of the present invention;

[0053] Figure 5 This is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. Detailed implementation manners

[0054] To make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below. Obviously, the described embodiments are only a part rather than all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope protected by the present invention.

[0055] Figure 1 This is a schematic structural diagram of a multi-spectral image intelligent simulation system based on a three-dimensional model provided by an embodiment of the present invention. Refer to Figure 1 , which specifically includes:

[0056] Three-dimensional modeling module: used to create a three-dimensional model of the object to be simulated.

[0057] Multi-spectral imaging module, connected to the three-dimensional modeling module, used to select different bands and image the three-dimensional model.

[0058] Import the three-dimensional model into ENVI, select visible light bands (red, green, blue) and near-infrared bands, adjust imaging parameters such as exposure time and resolution, start the imaging process, and perform multi-spectral imaging on the object to be simulated.

[0059] As Figure 2 shown, the multi-spectral imaging module includes the following sub-modules:

[0060] Lighting calculation sub-module: used to calculate the lighting intensity according to the light source position and light source intensity.

[0061] As Figure 3 shown, the lighting calculation sub-module includes the following units:

[0062] Light source positioning unit: used to determine the light source position and light source intensity of the light source.

[0063] Determination of the light source position and light source intensity of a light source, including: configuring a light source for a three-dimensional model, taking a dynamic image of the three-dimensional model with the configured light source, determining a reference image from the dynamic image, obtaining the pixel coordinates and average brightness value of the highlighted area of the reference image, and mapping the pixel coordinates to three-dimensional coordinates; using a line detection algorithm to detect the lines in the reference image, obtaining the line direction vectors based on the detected lines; obtaining the light source direction vector according to the line direction vectors; constructing an objective function, minimizing the objective function to obtain the light source position, and obtaining the distance between the light source and the highlighted area according to the light source position and the three-dimensional coordinates; obtaining the light source intensity according to the average brightness value and the distance between the light source and the highlighted area.

[0064] In this embodiment, in the process of determining the reference image from several dynamic images, first, evaluation indexes are determined: the symmetry, shape factor, and dynamic image contrast of the highlighted area. Calculate the evaluation indexes, calculate the comprehensive score based on the evaluation indexes, and determine the dynamic image of the highlighted area with the highest comprehensive score as the reference image. Specifically:

[0065] Calculation of the symmetry of the highlighted area of the dynamic image: First, perform brightness analysis on each dynamic image to extract the highlighted area, and use the threshold segmentation algorithm to convert the highlighted area into a binary highlighted image. Based on the binary highlighted image, perform two-dimensional Fourier transform to convert the spatial domain to the frequency domain, and move the zero-frequency component of the spectrum to the center position for better observation of the symmetry of the spectrum. Calculate the symmetry of the spectrum in the horizontal and vertical directions, perform horizontal and vertical flips on the spectrum, calculate the difference between the flipped spectrum and the original spectrum, and calculate the symmetry score. The smaller the score, the better the symmetry.

[0066] ;

[0067] In the formula, represents the frequency value in the frequency domain, represents the pixel value of the binary highlighted image in the spatial domain, M and N respectively represent the width and height of the image, j represents the imaginary unit, is the pi, .

[0068] ;

[0069] In the formula, S represents the symmetry score, represents the frequency domain value at the position symmetric about the horizontal axis of the image, represents the frequency domain value at the position symmetric about the vertical axis of the image.

[0070] In this embodiment, the value range of the symmetry score is from 1 to Q. The first threshold, the second threshold, and the third threshold are constructed according to the value range of the symmetry score, and the symmetry score rating is determined according to the first threshold, the second threshold, and the third threshold. Further, the first threshold is [1, Q / 4], the second threshold is (Q / 4, 3Q / 4], and the third threshold is (3Q / 4, Q]. When the symmetry score is within the range of the first threshold, the symmetry score rating is determined as c. When the symmetry score is within the range of the second threshold, the symmetry score rating is determined as h. When the symmetry score is within the range of the third threshold, the symmetry score rating is determined as w, where c > h > w.

[0071] The Fourier transform converts the image from the spatial domain to the frequency domain, and the frequency spectrum reflects the global features of the image. By evaluating the symmetry of the frequency spectrum, a more comprehensive understanding of the overall symmetry of the image can be obtained, rather than just the local symmetry.

[0072] Calculation of the shape factor of the highlighted area: The highlighted area that is closer to a circle can be determined as the reference image. Ellipse fitting is used to evaluate the shape factor. The closer the ratio of the major axis to the minor axis of the ellipse fitting is to 1, the closer the shape is to a circle.

[0073] ;

[0074] In the formula, L represents the shape factor, represents the length of the major axis of the fitted ellipse, represents the length of the minor axis of the fitted ellipse.

[0075] In this embodiment, a shape factor threshold is constructed, and the shape factor rating is determined according to the shape factor threshold. Preferably, the shape factor threshold is preferably 0.8. For a shape factor greater than or equal to 0.8, the shape factor rating is determined as 0.9. For a shape factor less than 0.8, the shape factor rating is determined as 0.3.

[0076] For the dynamic image contrast, the difference in the average luminance values between the highlighted area and the background area of the dynamic image is calculated. The greater the difference, the greater the contrast.

[0077] ;

[0078] In the formula, P represents the contrast, represents the average luminance value of the highlighted area, represents the average luminance value of the background area.

[0079] ;

[0080] SC represents the comprehensive score.

[0081] The comprehensive score can more accurately screen out the best reference image, improving the accuracy of subsequent image processing and analysis. The comprehensive score is based on objective mathematical calculations, reducing the subjective factors of human judgment and enhancing the reliability of the results.

[0082] Take the dynamic image with the maximum comprehensive score as the reference image of this embodiment, and calculate the average brightness value of the highlighted area of the reference image.

[0083] Furthermore, identify the edges of the reference image through an edge detection algorithm, and use the Hough transform to detect straight lines in the image. The Hough transform can transform points in the image space into curves in the parameter space, thereby finding straight lines. For each detected straight line, its two endpoints (x1, y1) and (x2, y2) can be obtained, and the straight line direction vector is calculated: .

[0084] For each straight line, its normal direction vector can be obtained by rotating 90 degrees. According to the incident direction, it can be determined whether the straight line direction vector is or . When multiple straight lines are detected, calculate the average value of all the normals of the straight lines as the overall light source direction vector.

[0085] In this embodiment, the preset light source position is (x, y, z), where z represents the height of the light source relative to the reference image plane. Construct an objective function to minimize the sum of the squares of the distances from each pixel point in all highlighted areas to the light source. If (x i , y i , z i ) is the position of the i-th pixel point, then the objective function is:

[0086] ; N represents the maximum number of pixel points in the highlighted area of the reference image. Use the gradient descent method, Newton's method or other numerical optimization methods to solve the minimum value of the objective function, thereby obtaining the light source position.

[0087] The significance of the design of this objective function is: find the light source position to minimize the sum of the squares of the distances from each pixel point in all highlighted areas to this light source position. That is, finding the minimum value of the objective function is the light source position.

[0088] Calculate the distance between each pixel point in the highlighted area of the reference image and the light source according to the Euclidean distance.

[0089] The calculation formula for the light source intensity is:

[0090] ;

[0091] In the formula, I represents the light source intensity, L represents the average brightness value of the highlighted area, and d represents the distance between the light source and the highlighted area.

[0092] A ray tracing unit for tracing the ray path from a light source to the surface of a three-dimensional model.

[0093] As Figure 4 shown, the ray tracing unit includes the following sub-units:

[0094] A ray initialization sub-unit for initializing one or more ray paths according to the light source position and the light source direction vector; determining the incident direction according to the light source direction vector.

[0095] The light source position obtained through the objective function is: w = (x s , y s , z s ), and the light source direction vector r = (r x , r y , r z ) is used as the starting point of the ray path to construct the path equation: R(t) = w + tr, where R(t) represents the end point of the path at a distance t along the ray direction. In this embodiment, the ray path is determined according to the path equation. In this embodiment, the incident direction is determined according to the light source direction vector r.

[0096] An intersection test sub-unit for calculating the intersection points of the ray path and the surface of the object to be simulated.

[0097] In practical applications, the surfaces of most three-dimensional models are complex polygon meshes. These meshes can represent various shapes, such as cars, people, buildings, etc. The advantage of polygon meshes is that they can flexibly represent any complex geometric shape, and can be decomposed into multiple simple triangles through triangulation, which is convenient for ray tracing and other geometric calculations.

[0098] Polygon meshes usually consist of multiple triangles, and each triangle has three vertices. To calculate the intersection points of the rays and the polygon meshes, this embodiment calculates the intersection points through the Möller–Trumbore algorithm.

[0099] Set three vertices of the preset polygon mesh as s0, s1, s2; calculate the triangle edge vectors: e1 = s1 - s0, e2 = s2 - s0. Specifically, e1 represents the vector from triangle vertex s0 to vertex s1, and e2 represents the vector from triangle vertex s0 to vertex s2. Calculate the auxiliary vector U: U = r × e2. This formula represents the cross product of the light source direction vector and the triangle edge vector. This auxiliary vector is perpendicular to the light source direction and one side of the triangle. Calculate the auxiliary vector T: T = w - s0. This formula represents the vector from the light source starting point w to triangle vertex s0. Calculate the auxiliary vector q: q = T × e1. This vector is also perpendicular to the light source direction and one side of the triangle. Calculate the determinant: a = e1 × U; a is a scalar used to check whether the light source is parallel or coincident with the triangle. In this embodiment, a threshold is preset , When , the light source is parallel or coincident with the triangle and there is no intersection point. If , the light source and the triangle have an intersection point. It is not enough to judge the intersection of the light source and the plane where the triangle is located only by the determinant a and the threshold. Even if the light ray intersects the plane, the intersection point may fall outside the triangle. In this case, the intersection point is not a valid intersection point because the light ray does not really intersect the triangle. For this reason, this embodiment further screens the determined intersection points: calculate the above intersection point (f, g, t) to judge whether the intersection point is inside the triangle: ; ; . Judge whether f is greater than or equal to 0 and less than or equal to 1, and at the same time, whether g is greater than or equal to 0 and less than or equal to 1, and at the same time , and at the same time 0 (representing that the intersection point is in the forward direction of the light source); if so (the above conditions must be met simultaneously), the intersection point is inside the triangle and is a valid intersection point. If not, the intersection point is outside the triangle and is an invalid intersection point. Finally, substitute into the light path equation to obtain the specific intersection point.

[0100] Attribute calculation subunit, used to calculate the intersection point attributes, and the intersection point attributes at least include the intersection point surface normal vector and the outgoing direction; obtain the half-way vector according to the incident direction and the outgoing direction, and obtain the incident angle according to the incident direction and the intersection point surface normal vector.

[0101] Intersection point surface normal vector is a vector perpendicular to the surface of the object to be simulated, pointing outward. For the triangle mesh, the intersection point surface normal vector is calculated by the cross product of two edge vectors of the triangle. The outgoing direction is the direction of the light source after reflection or refraction on the surface. The outgoing direction obtained according to the law of reflection is:

[0102] ;

[0103] or the exit direction obtained according to the law of refraction is:

[0104] , ;

[0105] and are the refractive indices of the incident medium and the exit medium, respectively.

[0106] The incident angle is the angle between the light source direction vector and the surface normal vector at the intersection point:

[0107] , .

[0108] The halfway vector refers to the average direction of the light source direction vector and the view direction vector. The view direction refers to the direction of the line connecting the position of the camera capturing the reference image and the intersection point.

[0109] Determine the incident light intensity of the light source reaching each point on the surface of the object to be simulated according to the light source intensity.

[0110] The calculation formula for the incident light intensity is:

[0111] ;

[0112] In the formula, I represents the light source intensity, represents the transmission factor, represents the attenuation factor.

[0113] The light intensity calculation unit is used to calculate the light intensity.

[0114] The light intensity is calculated based on the incident light intensity, the incident direction, the exit direction, and the surface normal vector at the intersection point.

[0115] ;

[0116] In the formula, represents the light intensity in the band , represents the band under which the light source direction vector is the incident light intensity of the light source reaching the i-th intersection point on the surface of the object to be simulated, represents the bidirectional reflectance distribution function, represents the band under which from the incident direction The incident light is in the exit direction The probability of exit, represents the surface normal vector of the i-th intersection point, represents in the incident direction The solid angle increment brought by the change in the upward direction, represents the incident direction, represents the exit direction.

[0117] In the above formula, actually refers to the solid angle increment brought by the tiny change in the incident direction . Here, refers to the incident direction, while corresponds to the tiny interval in this direction. When we calculate the integral, we are actually accumulating along all possible incident directions, and each accumulation considers the situation within a small direction interval, thus obtaining the overall effect within the entire incident direction range.

[0118] This formula is used to calculate the light intensity under a specific wavelength band . Its core lies in considering the influence of all possible incident directions on the final light intensity through integration. The following is a specific explanation of each component:

[0119] , indicates that under the wavelength band , the light intensity incident on a certain point on the surface from the direction . Here, represents any incident direction, while quantifies the amount of energy reaching the surface from this direction.

[0120] , called the bidirectional reflectance distribution function, which describes the probability density of light being reflected from the incident direction to the exit direction . This function takes into account the physical properties of the material surface, such as diffuse reflectivity and specular reflectivity, and varies with the wavelength, that is, varies with .

[0121] , this is the dot product of the incident direction vector and the surface normal vector , which reflects the angle of the incident light relative to the surface. When the incident light is perpendicular to the surface, and have an included angle of 0 degrees, and at this time their dot product is the largest; as the angle increases, the dot product decreases, meaning that the effective energy of the incident light also decreases.

[0122] : It refers to the solid angle element corresponding to a tiny interval in the incident direction. By integrating over all possible incident directions, we can comprehensively consider the contribution of light from different directions to the total light intensity.

[0123] This formula emphasizes that by considering the incident light intensity, bidirectional reflectance distribution function, and incident angle together, an accurate simulation of complex lighting environments can be achieved. This method can not only handle direct lighting but also take into account the indirect lighting effects after multiple reflections. At the same time, since both the incident light intensity and the bidirectional reflectance distribution function depend on the wavelength, this method is particularly suitable for the simulation of multispectral images because it can capture the differences in lighting characteristics at different wavelengths. In addition, the use of the bidirectional reflectance distribution function enables this method to adapt to various material properties, thus being applicable to a wide range of material types. Ultimately, this physically based rendering method can significantly improve the realism of the synthesized images, especially in the representation of complex lighting conditions and fine material details. In summary, this formula provides a systematic solution for calculating the light intensity at a specific band, combining the advantages of physical principles and mathematical models, and laying the foundation for high-precision multispectral image simulation.

[0124] ;

[0125] In the formula, represents the diffuse reflection coefficient, represents the specular reflection coefficient, represents the band under the diffuse reflectance, represents the Fresnel term, represents the micro-surface distribution function, represents the geometric occlusion function, represents the incident angle, h represents the half-way vector, represents pi, represents the band under the diffuse reflection coefficient, represents the band under the specular reflection coefficient, represents the incident direction from the incident light in the outgoing direction of the geometric occlusion function.

[0126] This formula is constructed based on physical principles and can accurately simulate the optical properties of different materials, including diffuse reflection and specular reflection. By introducing the micro-surface distribution function and the geometric occlusion function, this formula can finely capture the influence of surface micro-structures, thereby significantly enhancing the realism of rendering. Specifically, the micro-surface distribution function describes the impact of surface micro-irregularities on light reflection, while the geometric occlusion function takes into account the occlusion effect between the incident light and the outgoing light. In addition, the Fresnel term further enhances the accurate simulation of the reflectivity changes at different incident angles. These factors work together, enabling this formula to not only handle multi-directional reflections in complex lighting environments but also accurately calculate the reflection probability of the light source in a specific direction at different wavelengths 𝜏τ. Therefore, this formula provides a comprehensive and accurate method for calculating the reflection probability of the light source in a specific direction, which is of great significance for enhancing realistic rendering in computer graphics, especially when dealing with high-precision materials and complex lighting conditions.

[0127] ;

[0128] In the formula, represents the roughness coefficient, represents the angle between the half-way vector and the surface normal vector at the intersection point.

[0129] Use image processing techniques to analyze the texture features in the reference image and obtain the texture feature values. Take the mean value of the texture feature values and use the normalized value as the roughness coefficient.

[0130] By considering the angle between the half-way vector and the surface normal vector at the intersection point, the micro-structure of the surface of the object to be simulated can be more accurately modeled, thereby generating a more realistic visual effect. This helps to create a material surface with rich details and a sense of layering. The micro-surface distribution function takes into account the degree of micro-concavity and convexity of the surface, which is particularly important for simulating specular reflection. The roughness coefficient controls the smoothness of the surface, thus affecting the intensity and range of specular reflection. Using the micro-surface distribution function, a more natural transition effect can be produced in the highlight area. This is because the micro-surface distribution function takes into account the influence of the surface micro-structure on light scattering, thus avoiding the sharp edges and unnatural highlight shapes that may occur in traditional models. By processing various types of surfaces in a unified way, whether smooth or rough, consistent results can be obtained. This helps to simplify the material design process and reduce the debugging time.

[0131] ;

[0132] In the formula, represents taking the minimum value.

[0133] Such a design enables the geometric occlusion function to effectively reflect the occlusion effect suffered by the light source when reaching the surface. Specifically, this function determines whether the light is occluded or partially occluded based on the relationship between the incident direction and the outgoing direction and the surface normal at the intersection point. If the dot product result of any direction and the surface normal at the intersection point is less than or equal to zero, that is, the angle formed by the light source and the surface is greater than 90 degrees, then that direction is considered completely occluded, and the value of the corresponding geometric occlusion function is 0. Conversely, if both directions are orthogonal or nearly orthogonal to the normal, it indicates that there is no obvious occlusion phenomenon, and at this time the function value is close to 1. In addition, through the minimum operation, even if there is slight occlusion in only one of the directions, it will cause an increase in the overall occlusion level, thus ensuring that all potential occlusion factors can be correctly reflected in any case. This design method is not only simple and intuitive, but also can largely retain the original advantages while increasing the sensitivity to local environmental information, so it is very suitable for rapid and accurate rendering calculations in complex lighting environments.

[0134] A rendering sub-module for rendering a multi-spectral image according to the illumination intensity.

[0135] Using the Shader programming ability in a physically based rendering engine, develop a multi-spectral rendering Shader. This Shader can receive different light source intensities as inputs and adjust the color response of the material according to light of different wavelengths, so as to render a multi-spectral image.

[0136] An image synthesis sub-module for synthesizing images formed by light sources of different bands into a single multi-spectral image.

[0137] Write a script using Python combined with the OpenCV or PIL library to read single-channel grayscale images of multiple bands, and then merge them into a multi-channel multi-spectral image according to specific synthesis rules (such as weighted average or physically based methods).

[0138] A physical property setting module, connected to the 3D modeling module, for defining physical properties of the surface of the object to be simulated in the 3D model; the physical properties at least include diffuse reflectivity and specular reflectivity.

[0139] Create a plugin in 3D modeling software to allow users to specify different physical properties for each part of the model, such as diffuse reflectivity, specular reflectivity, etc. These properties can be directly set in the material editor and saved to an external file through the export function for use by subsequent simulation modules.

[0140] A simulation module: connected to the multi-spectral imaging module and the physical property setting module, for generating multi-spectral images.

[0141] Build a high-performance computing framework based on CUDA or OpenCL, which can load the material information provided by the physical property setting module and perform simulation calculations of multi-spectral images in combination with lighting conditions. During the simulation process, the interaction between light and matter can be simulated according to different lighting angles and intensities, and finally high-quality multi-spectral images are output.

[0142] Multi-source heterogeneous data acquisition module: Connected to the simulation module and the 3D modeling module, it is used to obtain RGB images and depth images according to multi-spectral images, and obtain 3D model data and environmental data according to the object to be simulated.

[0143] Develop a set of hardware interface programs that can collect data from various sensors (such as RGB cameras, depth cameras, multi-spectral cameras, etc.) and transmit this data to the simulation module and the 3D modeling module through a unified data format.

[0144] Analysis module, connected to the multi-source heterogeneous data acquisition module, is used to perform 3D reconstruction based on multi-spectral images, RGB images, depth images, 3D model data, and environmental data to obtain simulation results.

[0145] Use machine learning algorithms (such as convolutional neural network CNN) to analyze the obtained multi-spectral images, RGB images, depth images, and environmental data to achieve 3D reconstruction. When training the model, a supervised learning method can be adopted, using known 3D scenes as labels to optimize the model performance. After training is completed, the model can automatically reconstruct a high-precision 3D model according to the input multi-source heterogeneous data.

[0146] Figure 5 It is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. As Figure 5 shown, the electronic device 500 includes one or more processors 501 and a memory 502.

[0147] The processor 501 can be a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and can control other components in the electronic device 500 to perform desired functions.

[0148] The memory 502 may include one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory, etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage media, and the processor 501 may run the program instructions to implement a multi-spectral image intelligent simulation system based on a three-dimensional model according to any embodiment of the present application as described above and / or other desired functions. Various contents such as initial external parameters, thresholds, etc. may also be stored in the computer-readable storage media.

[0149] In one example, the electronic device 500 may further include: an input device 503 and an output device 504, and these components are interconnected through a bus system and / or other forms of connection mechanisms (not shown). The input device 503 may include, for example, a keyboard, a mouse, etc. The output device 504 may output various information to the outside, including early warning prompt information, braking force, etc. The output device 504 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0150] Of course, for simplicity, Figure 5 only some of the components related to the present application in the electronic device 500 are shown, and components such as buses, input / output interfaces, etc. are omitted. In addition, according to specific application scenarios, the electronic device 500 may further include any other appropriate components.

[0151] In addition to the above methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, and when the computer program instructions are run by a processor, the processor is caused to execute the steps of a multi-spectral image intelligent simulation system based on a three-dimensional model provided by any embodiment of the present application.

[0152] The computer program product may be written in any combination of one or more programming languages to write program code for performing the operations of the embodiments of the present application. The programming languages include object-oriented programming languages such as Java, C++, etc., and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's device, executed as an independent software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0153] In addition, an embodiment of the present application may also be a computer-readable storage medium storing computer program instructions, which, when run by a processor, cause the processor to execute the steps of a multi-spectral image intelligent simulation system based on a three-dimensional model provided by any embodiment of the present application.

[0154] The computer-readable storage medium may employ any combination of one or more readable media. The readable media may be a readable signal medium or a readable storage medium. The readable storage medium may, for example, include but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the readable storage medium (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0155] It should be noted that the terms used in the present invention are only for describing specific embodiments and do not limit the scope of the present application. As shown in the specification of the present invention, unless the context clearly indicates an exception, words such as "a", "an", "one", and / or "the" do not specifically refer to the singular and may also include the plural. The term "comprising", "including", or any other variant thereof is intended to cover a non-exclusive inclusion, such that a process, method, or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, or device. Without further limitation, an element defined by the statement "comprising one..." does not exclude the presence of additional identical elements in the process, method, or device comprising the element.

[0156] It should also be noted that the orientation or positional relationship indicated by terms such as "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus cannot be understood as a limitation of the present invention. Unless otherwise clearly specified and limited, terms such as "installed", "connected", "coupled", etc. should be understood in a broad sense. For example, it may be a fixed connection, a detachable connection, or an integral connection; it may be a mechanical connection or an electrical connection; it may be directly connected or indirectly connected through an intermediate medium, and it may be the internal communication of two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0157] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. An intelligent simulation system for multispectral images based on a three-dimensional model, characterized in that It includes the following modules: 3D modeling module: used to create a 3D model of the object to be simulated; Multispectral imaging module, connected to the 3D modeling module, used to select light sources of different bands and image the 3D model; Multispectral imaging module, including the following sub-modules: Lighting calculation sub-module, used to calculate the lighting intensity according to the light source position and light source intensity; Lighting calculation sub-module, including the following units: Light source positioning unit, used to determine the light source position and light source intensity of the light source; Configure a light source for the 3D model, take a dynamic image of the 3D model with the configured light source, determine a reference image from the dynamic image, and obtain the pixel coordinates and average brightness value of the highlighted area of the reference image; Determine evaluation indicators: symmetry, shape factor, and dynamic image contrast of the highlighted area; calculate the evaluation indicators, calculate a comprehensive score based on the evaluation indicators, and determine the dynamic image of the highlighted area with the highest comprehensive score as the reference image; Perform a two-dimensional Fourier transform on the highlighted area of each dynamic image, convert the spatial domain to the frequency domain, and move the zero-frequency component of the spectrum to the center position; calculate the symmetry of the spectrum in the horizontal and vertical directions, flip the spectrum horizontally and vertically, calculate the difference between the flipped spectrum and the original spectrum, and calculate the symmetry score; ； Where S represents the symmetry score, represents the frequency domain value of the image at the position symmetric about the horizontal axis, represents the frequency domain value of the image at the position symmetric about the vertical axis, represents the frequency domain value, represents the pixels of the binary highlighted image in the spatial domain, and M and N respectively represent the width and height of the image; The value range of the symmetry score is 1-Q. Construct a first threshold, a second threshold, and a third threshold according to the value range of the symmetry score. The first threshold is [1, Q / 4], the second threshold is (Q / 4, 3Q / 4], and the third threshold is (3Q / 4, Q]. When the symmetry score is within the first threshold range, determine the score of this symmetry score as c. When the symmetry score is within the second threshold range, determine the score of this symmetry score as h. When the symmetry score is within the third threshold range, determine the score of this symmetry score as w, where c>h>w; Use ellipse fitting to evaluate the shape factor: ; L represents the shape factor, represents the major axis length of the fitted ellipse, represents the minor axis length of the fitted ellipse; Construct a shape factor threshold, and determine the shape factor score according to the shape factor threshold. The shape factor threshold is 0.

8. For a shape factor greater than or equal to 0.8, determine the shape factor score as 0.

9. For a shape factor less than 0.8, determine the shape factor score as 0.3; Dynamic image contrast, by calculating the difference in the average brightness value between the highlighted area and the background area of the dynamic image; ； Wherein, P represents the contrast ratio, represents the average luminance value of the highlighted area, represents the average luminance value of the background area; ; SC represents the comprehensive score; Ray tracing unit, used to trace the light path from the light source to the surface of the 3D model; Lighting intensity calculation unit, used to calculate the lighting intensity; Rendering sub-module, used to render a multispectral image according to the lighting intensity; Image synthesis sub-module, used to synthesize the images of light sources of different bands into a single multispectral image; Physical property setting module, connected to the 3D modeling module, used to define physical properties for the surface of the object to be simulated in the 3D model; the physical properties include diffuse reflectivity and specular reflectivity; Simulation module: connected to the multispectral imaging module and the physical property setting module, used to generate a multispectral image; Multi-source heterogeneous data acquisition module: connected to the simulation module and the 3D modeling module, used to obtain an RGB image and a depth image according to the multispectral image, and obtain 3D model data and environmental data according to the object to be simulated; The analysis module, connected to the multi-source heterogeneous data acquisition module, is used to perform 3D reconstruction based on multi-spectral images, RGB images, depth images, 3D model data, and environmental data to obtain simulation results.

2. The multispectral image intelligent simulation system based on a 3D model according to claim 1, characterized in that Determination of the light source position and light source intensity of the light source, including: mapping pixel coordinates to 3D coordinates; detecting straight lines in the reference image using a straight line detection algorithm, and obtaining the straight line direction vector according to the detected straight lines; obtaining the light source direction vector according to the straight line direction vector; constructing an objective function, minimizing the objective function to obtain the light source position, and obtaining the distance between the light source and the highlighted area according to the light source position and 3D coordinates; obtaining the light source intensity according to the average brightness value and the distance between the light source and the highlighted area.

3. The multispectral image intelligent simulation system based on a 3D model according to claim 2, wherein, The calculation formula for the light source intensity is: ； In the formula, I represents the light source intensity, L represents the average brightness value of the highlighted area, and d represents the distance between the light source and the highlighted area.

4. An intelligent simulation system for multispectral images based on a three-dimensional model according to claim 3, characterized in that, The ray tracing unit includes the following sub-units: The ray initialization sub-unit is used to initialize one or more ray paths according to the light source position and the light source direction vector; determine the incident direction according to the light source direction vector; The intersection test sub-unit is used to calculate the intersection points of the ray paths and the surface of the object to be simulated; The property calculation sub-unit is used to calculate the intersection point properties, and the intersection point properties at least include the surface normal vector of the intersection point and the outgoing direction; obtain the half-way vector according to the incident direction and the outgoing direction, and obtain the incident angle according to the incident direction and the surface normal vector of the intersection point.

5. An intelligent simulation system for multi-spectral images based on a 3D model according to claim 4, characterized in that, Determine the incident illumination intensity of the light source reaching each point on the surface of the object to be simulated according to the light source intensity; The illumination intensity is calculated according to the incident illumination intensity, the incident direction, the outgoing direction, and the surface normal vector of the intersection point.

6. The intelligent simulation system of multispectral images based on a three-dimensional model according to claim 5, characterized in that, The calculation formula for the illumination intensity is: ； In the formula, represents the light intensity in the band, represents the incident light intensity of the light source with the light source direction vector reaching the i-th intersection point on the surface of the object to be simulated, represents the bidirectional reflectance distribution function, represents the probability that the light incident from the incident direction is emitted in the emission direction in the band, represents the surface normal vector of the i-th intersection point, represents the solid angle increment brought by the direction change in the incident direction represents the incident direction, represents the emission direction.

7. An intelligent simulation system for multi-spectral images based on a three-dimensional model according to claim 6, characterized in that The calculation formula for the incident illumination intensity is: ； Wherein, I represents the light source intensity, represents the transmission factor, represents the attenuation factor.

8. An intelligent simulation system for multi-spectral images based on a three-dimensional model according to claim 7, characterized in that, The bidirectional reflectance distribution function is: ； In the formula, represents the diffuse reflection coefficient, represents the specular reflection coefficient, represents the wavelength band and the diffuse reflectance under it, represents the Fresnel term, represents the micro-surface distribution function, represents the geometric shadowing function, represents the angle of incidence, h represents the half-way vector, represents pi, represents the wavelength band and the diffuse reflection coefficient under it, represents the wavelength band and the specular reflection coefficient under it, represents the geometric shadowing function of the light incident from the incident direction in the outgoing direction .

9. The intelligent simulation system for multi-spectral images based on a three-dimensional model according to claim 8, characterized in that, The micro-surface distribution function is: ； In the formula, represents the roughness coefficient, represents the angle between the half-range vector and the surface normal vector of the intersection point.

10. The intelligent simulation system of multi-spectral images based on a 3D model according to claim 8, characterized in that, The geometric occlusion function is: ； In the formula, represents taking the minimum value.

Images