Spectral reflectivity reconstruction method and multispectral image synthesis method, device and system
By multiple methods of projecting light from different bands and taking images, the spectral reflectivity function of the target object is calculated, which solves the problem that multi-spectral image processing in the prior art cannot reflect the imaging characteristics of the object, and achieves high spatial and high spectral resolution Take into account both.
Patent Information
- Application Number
- CN202410981644.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-12-06
Smart Images

Figure CN119090982A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a spectral reflectance reconstruction method and a multi-spectral image synthesis method, device and system. Background Art
[0002] Hyperspectral imaging technology can accurately identify the substances in the scene, greatly expanding the capabilities of remote sensing technology, and playing an important role in geological surveys, crop monitoring, national defense and security. When the number of bands increases, due to the limited solar irradiance, the number of solar photons scattered on each band of the hyperspectral image (HSI) decreases accordingly. Therefore, the spatial resolution of HSI is usually low. Compared with HSI, multispectral image (MSI) has fewer bands and higher spatial resolution. In order to obtain HSI with high spatial resolution and high spectral resolution, the fusion of HSI and MSI (FHM) is a typical strategy.
[0003] Multispectral images are increasingly being used in defect detection, palmprint recognition, multimodal data analysis, and other fields. Multispectral images are often obtained through image synthesis, which is a processing technique that converts multi-spectral black-and-white images into color images through color synthesis of multispectral images. The purpose of image synthesis is to enhance information such as texture and contrast on the image. Different color synthesis schemes can be implemented for different application purposes. Commonly used image synthesis methods include color synthesis and pseudo-color synthesis. The above-mentioned image synthesis methods do not take into account the characteristics of the light source or reflector itself, and directly use image data for synthesis analysis and processing, which cannot reflect the imaging characteristics of the object itself. Summary of the invention
[0004] The present invention provides a spectral reflectance reconstruction method and a multi-spectral image synthesis method, device and system, as well as a computer-readable storage medium, aiming to better reflect the imaging characteristics of an object itself.
[0005] According to the first aspect, an embodiment provides a multispectral image synthesis method, comprising:
[0006] Control the multi-spectral light source to project monochromatic light onto the target object, projecting M times in total, projecting monochromatic light of different wavelengths onto the target object each time; triggering the camera to capture an image of the target object after each projection, and obtaining M N-channel images after M captures, where N is the number of channels of the camera, and both M and N are integers greater than 1;
[0007] Calculating a spectral reflectance function of the target object according to a spectral curve of the multi-spectral light source, a spectral response curve of the camera, and the M N-channel images;
[0008] The spectral reflectance function is used to calculate the reflectance of each pixel point in the multispectral image of the target object to multiple different spectral bands, and the reflectance of each pixel point to multiple different spectral bands is used as the pixel value of the pixel point to obtain the multispectral image of the target object.
[0009] According to the second aspect, an embodiment provides a spectral reflectance reconstruction method for obtaining a spectral reflectance function of a target object, the spectral reflectance reconstruction method comprising:
[0010] Control the multi-spectral light source to project monochromatic light onto the target object, projecting M times in total, projecting monochromatic light of different wavelengths onto the target object each time; triggering the camera to capture an image of the target object after each projection, and obtaining M N-channel images after M captures, where N is the number of channels of the camera, and both M and N are integers greater than 1;
[0011] The spectral reflectance function of the target object is calculated according to the spectral curve of the multi-spectral light source, the spectral response curve of the camera and the M N-channel images.
[0012] According to a third aspect, an embodiment provides a multispectral image synthesis device, comprising:
[0013] A single-band image shooting module is used to control the multi-spectral light source to project monochromatic light onto the target object, projecting a total of M times, each time projecting monochromatic light of a different band onto the target object, and triggering the camera to shoot the image of the target object after each projection, and obtaining M N-channel images after M shots, where N is the number of channels of the camera, and M and N are both integers greater than 1;
[0014] A spectral reflectance function determination module, connected to the single-band image shooting module, for calculating the spectral reflectance function of the target object according to the spectral curve of the multi-spectral light source, the spectral response curve of the camera and the M N-channel images;
[0015] The multispectral image synthesis module is connected to the spectral reflectance function determination module, and is used to calculate the reflectance of each pixel point in the multispectral image of the target object to multiple different spectral bands using the spectral reflectance function, and use the reflectance of each pixel point to multiple different spectral bands as the pixel value of the pixel point to obtain the multispectral image of the target object.
[0016] According to a fourth aspect, an embodiment provides a spectral reflectance reconstruction device for obtaining a spectral reflectance function of a target object, the spectral reflectance reconstruction device comprising:
[0017] A single-band image shooting module is used to control the multi-spectral light source to project monochromatic light onto the target object, projecting a total of M times, each time projecting monochromatic light of a different band onto the target object, and triggering the camera to shoot the image of the target object after each projection, and obtaining M N-channel images after M shots, where N is the number of channels of the camera, and M and N are both integers greater than 1;
[0018] The spectral reflectance function determination module is connected to the single-band image shooting module and is used to calculate the spectral reflectance function of the target object according to the spectral curve of the multi-spectral light source, the spectral response curve of the camera and the M N-channel images.
[0019] According to a fifth aspect, an embodiment provides a multispectral image synthesis system, characterized in that it includes:
[0020] Multi-spectral light source, used to project monochromatic light in different wavelength bands;
[0021] A camera, used to capture images of the target object;
[0022] A display, used to display the multispectral image of the target object;
[0023] A controller, connected to the multi-spectral light source, the camera and the display, is configured to perform the following steps:
[0024] Control the multi-spectral light source to project monochromatic light onto the target object M times in total, projecting monochromatic light of different wavelength bands onto the target object each time; trigger the camera to capture an image of the target object after each projection, and obtain M N-channel images after M captures, where N is the number of channels of the camera, and both M and N are integers greater than 1;
[0025] Calculating a spectral reflectance function of the target object according to a spectral curve of the multi-spectral light source, a spectral response curve of the camera, and the M N-channel images;
[0026] The spectral reflectance function is used to calculate the reflectance of each pixel point in the multispectral image of the target object to multiple different spectral bands, and the reflectance of each pixel point to multiple different spectral bands is used as the pixel value of the pixel point to obtain the multispectral image of the target object;
[0027] The multispectral image of the target object is output to the display for display.
[0028] According to the sixth aspect, an embodiment provides a computer-readable storage medium having a program stored thereon, the program being executable by a processor to implement the multispectral image synthesis method described in the first aspect or the spectral reflectance reconstruction method described in the second aspect.
[0029] According to the spectral reflectance reconstruction method and the multispectral image synthesis method, device and system of the above-mentioned embodiments, a multispectral light source is used to project monochromatic light of different bands to the target object multiple times, and a camera is triggered to capture an image of the target object after each projection, thereby obtaining a corresponding N-channel image corresponding to each band, wherein N is the number of channels of the camera, and then the spectral reflectance function of the target object can be calculated based on the spectral curve of the multispectral light source, the spectral response curve of the camera and the multiple N-channel images obtained, which can better reflect the imaging characteristics of the object; the spectral reflectance function of the target object is used to calculate the reflectance of each pixel point to multiple different spectral bands as the pixel value of the pixel point, to form a multispectral image of the target object, so that the obtained multispectral image can more realistically reflect the imaging effect of the object itself. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 A schematic diagram of the structure of a multi-spectral image synthesis system according to an embodiment;
[0031] Figure 2 A flowchart of a spectral reflectance reconstruction method according to an embodiment;
[0032] Figure 3 is a spectral reflectance function reconstructed in one embodiment;
[0033] Figure 4 A flowchart of a multispectral image synthesis method according to an embodiment;
[0034] Figure 5 A flowchart of calibrating a spectral response curve of a camera in one embodiment;
[0035] Figure 6 The spectral response curve of a camera is calibrated;
[0036] Figure 7 A flowchart of establishing a grayscale-optical power lookup table in an embodiment;
[0037] Figure 8 This is the spectral curve of a calibrated DLP projector;
[0038] Fig. 9 A flowchart of a flat field correction according to an embodiment;
[0039] Fig.10 A schematic diagram of the structure of a spectral reflectance reconstruction device according to an embodiment;
[0040] Fig.11 A schematic diagram of the structure of a multi-spectral image synthesis device according to an embodiment;
[0041] Fig.12 FIG. 4 is a schematic diagram of the structure of a multi-spectral image synthesis device according to another embodiment. DETAILED DESCRIPTION
[0042] The present invention is further described in detail below by specific embodiments in conjunction with the accompanying drawings. Wherein similar elements in different embodiments adopt associated similar element numbers. In the following embodiments, many detailed descriptions are for making the present application better understood. However, those skilled in the art can easily recognize that some features can be omitted in different situations, or can be replaced by other elements, materials, methods. In some cases, some operations related to the present application are not shown or described in the specification, this is to avoid the core part of the present application being overwhelmed by too much description, and for those skilled in the art, it is not necessary to describe these related operations in detail, and they can fully understand the related operations according to the description in the specification and the general technical knowledge in the art.
[0043] In addition, the features, operations or characteristics described in the specification can be combined in any appropriate manner to form various implementations. At the same time, the steps or actions in the method description can also be interchanged or adjusted in a manner that is obvious to those skilled in the art. Therefore, the various sequences in the specification and the drawings are only for the purpose of clearly describing a certain embodiment and are not meant to be a required sequence, unless otherwise specified that a certain sequence must be followed.
[0044] The serial numbers of the components in this document, such as "first", "second", etc., are only used to distinguish the objects described and do not have any order or technical meaning. The "connection" and "coupling" mentioned in this application, unless otherwise specified, include direct and indirect connections (couplings).
[0045] The research goal of multispectral imaging is to restore the true color of the surface of an object. It is widely used in industrial inspection and scientific research, and is a field that has been increasingly studied in China recently. There are generally two types of light sources used in the current mainstream methods. One is to use LEDs of different bands for multiple illuminations, and the other is to use color filters of different bands for multiple illuminations. Among them, the method of using LEDs of different bands is more convenient to operate and acquire images faster, but it is necessary to ensure that multiple LEDs of different bands can irradiate the object in a consistent direction. The general method is to mix multiple LED lamp beads of different bands that are inlaid with each other, but this light source needs to be specially made and is not easy to obtain.
[0046] In recent years, multispectral fusion has been mainly processed using the following methods:
[0047] (1) Methods based on deep convolutional neural networks (CNNs): The residual learning method based on deep CNNs directly learns image priors and then integrates the learned priors into the HSI and MSI fusion framework. Although the deep learning fusion framework has certain advantages, it requires a large number of images to train specialized deep learning models for different types of sensors, thus sacrificing flexibility.
[0048] (2) Model-based methods: Model-based methods establish mathematical models from the perspective of tensors and use some tensor decomposition methods to perform multispectral fusion. This type of method relies heavily on the establishment of mathematical models and optimization algorithms and cannot adapt to the image fusion requirements of different bands and scenes.
[0049] (3) Extension of panchromatic sharpening method: The HSI spectrum is divided into several regions, and the traditional panchromatic sharpening technology is used to fuse the HSI in each spectral region, and the HSI is estimated by optimizing the fusion model; this type of method establishes a fusion model, including fidelity and regularization terms based on spatial (spectral) degradation mechanisms and prior information, and uses effective solutions to solve these models. Therefore, the model-based method can achieve a balance between spatial and spectral quality. This method depends on the granularity of the spectral region division. The smaller the granularity of the division, the higher the algorithm accuracy, but the slower the speed, and vice versa.
[0050] Based on the advantages and disadvantages of the prior art, the inventor has proposed a multispectral image synthesis system, method and spectral reflectance reconstruction method to improve the effect of multispectral imaging of objects. The multispectral image synthesis system provided by the present invention is first introduced below. Figure 1 The multi-spectral image synthesis system provided by the present invention includes a multi-spectral light source 11, a camera 12, a display 13 and a controller 14.
[0051] The multi-spectral light source 11 is used to project monochromatic light of different wavelength bands, that is, the multi-spectral light source 11 has the ability to project light or images of different colors, and specifically can be various lighting lamps, projectors, and the like.
[0052] In some embodiments of the present invention, the multi-spectral light source 11 uses a DLP (Digital Light Processing) projector. DLP projectors have the advantages of high frame rate, high brightness, and easy availability. In addition, DLP color projectors use a multi-LED light path multiplexing mode, so there is no need to worry about inconsistent lighting directions.
[0053] The DLP projector used in the present invention may be a DLP color projector, and the DLP color projector may not be limited to the commonly used TI (Texas Instruments) DLP projector on the market, but the LED lamp beads used should be single-band as much as possible and the bands of lamp beads of different colors should not be the same. Generally, mainstream DLP color projectors include three lamp beads of three different colors of RGB, which can project three colors of monochromatic light. Of course, the present invention is not limited to this implementation method. The DLP projector used in the present invention may also be a DLP white projector. By adding a color filter / color wheel to the DLP white projector and filtering the projected white light, the effect of producing light of different colors can also be achieved.
[0054] It is understandable that a DLP projector can have multiple monochrome channels of different wavelengths. For example, a DLP projector including three lamp beads of three different colors, RGB, has three monochrome channels, and the purpose of projecting monochrome light can be achieved by making one of the monochrome channels of the DLP projector perform projection. Figure 1 bit can achieve a frame rate of over 1,000 frames, has an extremely high frame rate upper limit, and can quickly restore the spectral reflectance information and multispectral images of the object surface at an extremely high frame rate.
[0055] The camera 12 is used to capture an image of a target object. The camera 12 may be a CCD camera, a CMOS camera, etc. It is understood that the camera 12 may have multiple channels, for example, a commonly used color camera has three channels of RGB.
[0056] The display 13 is used to display the multispectral image of the target object. The display 13 can be an LCD display, an LED display, etc.
[0057] The controller 14 is used to execute the spectral reflectance reconstruction method of the present invention, or execute the multispectral image synthesis method of the present invention to obtain the multispectral image of the target object, and output the multispectral image of the target object to the display 13 for display. The controller 14 can be a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other general controllers such as a microprocessor, etc., and can also be discrete gates or transistor logic devices, discrete hardware components, etc.
[0058] The spectral reflectance reconstruction method of the present invention is introduced below. The spectral reflectance reconstruction method is used to obtain the spectral reflectance function of the target object and reconstruct the spectral reflectance of the object. Please refer to Figure 2 In some embodiments, the spectral reflectance reconstruction method includes steps 110 to 120, which are described in detail below.
[0059] Step 110: Control the multi-spectral light source to project monochromatic light onto the target object, for a total of M times, each time projecting monochromatic light of a different wavelength band onto the target object; after each projection, trigger the camera to capture an image of the target object, and after M shots, obtain M N-channel images, where N is the number of channels of the camera, and M and N are both integers greater than 1.
[0060] In some embodiments, the multi-spectral light source uses a DLP projector, M is the number of channels of the DLP projector, and in step 110, the multi-spectral light source is controlled to project monochromatic light onto the target object, a total of M times, and each time monochromatic light of a different band is projected onto the target object. Specifically includes: controlling the DLP projector to project a monochromatic image onto the target object, a total of M times, and each time using a different monochromatic channel to project the monochromatic image onto the target object.
[0061] Step 120: Calculate the spectral reflectance function of the target object according to the spectral curve of the multi-spectral light source, the spectral response curve of the camera, and the M N-channel images.
[0062] The light beam emitted by the multi-spectral light source is reflected by the target object and then enters the camera for imaging. According to the relevant optical imaging principle, after the multi-spectral light source irradiates the object, the image formed by the camera can be determined according to the spectral curve of the multi-spectral light source, the spectral reflectance function of the object, and the spectral response curve of the camera. Those skilled in the art can construct an imaging model to represent the relationship between M N-channel images taken by the camera and the spectral curve of the multi-spectral light source, the spectral reflectance function of the target object, and the spectral response curve of the camera according to the actual imaging situation, so as to solve the spectral reflectance function of the target object according to this imaging model.
[0063] In some embodiments of the present invention, one or more reflectance basis functions are preset, and the spectral reflectance function is determined by the preset reflectance basis function and its coefficients. Then step 120 specifically includes: calculating the coefficients of the reflectance basis function according to the spectral curve of the multi-spectral light source, the reflectance basis function, the spectral response curve of the camera and M N-channel images; determining the spectral reflectance function according to the reflectance basis function and the coefficients of the reflectance basis function.
[0064] Since the relationship between the M N-channel images taken by the camera and the spectral curve of the multi-spectral light source, the spectral reflectance function of the target object, and the spectral response curve of the camera can be represented by a mathematical model, and the spectral reflectance function is determined according to the reflectance basis function and its coefficients, the relationship between the M N-channel images taken and the spectral curve of the multi-spectral light source, the reflectance basis function and its coefficients, and the spectral response curve of the camera can be obtained, and the coefficients of the reflectance basis function can be solved accordingly. By simplifying the spectral reflectance function to the reflectance basis function and its coefficients for representation, since the reflectance basis function is known, it is only necessary to solve the coefficients when solving, which simplifies the solution process and improves the calculation efficiency.
[0065] M×N 1-channel images can be obtained from M N-channel images taken by the camera. In some embodiments, there are multiple reflectance basis functions, so the relationship between the coefficients of the reflectance basis function, the spectral curve of the multi-spectral light source, the reflectance basis function, the spectral response curve of the camera and the M×N 1-channel images can be pre-constructed, and the coefficient vector α is obtained by solving, where I is a matrix composed of M×N 1-channel images. It can be understood that F is a matrix constructed based on the spectral curve of the multi-spectral light source, the reflectance basis function and the spectral response curve of the camera. Therefore, in this embodiment, the M N-channel images are first converted into M×N 1-channel images, and then the coefficient vector α is obtained by solving according to the relationship I=Fα.
[0066] In some embodiments, there are 8 reflectance basis functions, the spectral curve of the multi-spectral light source includes spectral curves of M types of projected monochromatic light, the spectral response curve of the camera includes spectral response curves of N channels of the camera, and the elements of the matrix F are determined by the following formula:
[0067] F m×n,j =∫l m (λ)b j (λ)c n (λ)dλ,
[0068] Where, m = 1, 2 ... M, n = 1, 2 ... N, j = 1, 2 ... 8, F m×n,j represents the element in the m×nth row and jth column of the matrix F, λ represents the wavelength of light, l m (λ) represents the spectrum curve of the mth monochromatic light, c n (λ) represents the spectral response curve of the nth channel of the camera, b j (λ) represents the jth reflectivity basis function.
[0069] In some embodiments, each monochromatic light corresponds to a spectral reflectance function, and there are a total of M spectral reflectance functions, each of which is determined by 8 reflectance basis functions. m×n,jWhen , the reflectance basis function corresponding to the spectral reflectance function of the mth monochromatic light is used.
[0070] Since common multi-spectral light sources and color cameras both have 3 channels, F is a 9×8 matrix, so the process of solving the coefficient vector α is overdetermined. In order to solve the coefficient vector α in this case, the present invention proposes a method of solving the coefficient vector α by using the properties of continuity and constant non-negative reflectivity, that is, the coefficient vector α is obtained by solving the following formula:
[0071]
[0072] where α * Represents the coefficient vector α, b obtained by the final solution m (λ) is an M×8 matrix, each row of which is the 8 reflectance basis functions of a spectral reflectance function. Indicates b m (λ) is the derivative with respect to wavelength λ, ||·||2 represents the L2 norm, and γ is the set coefficient.
[0073] In the objective function above, the error between the estimated grayscale value and the grayscale value of the captured image |I-Fα| 2 As the first item, it is hoped that the error between the estimated grayscale value and the grayscale value of the captured image is minimized. In order to increase the constraint expression, the inventor also added a second item By calculating b m (λ) For the rate of change of wavelength, a regularized L2 constraint condition is established based on the dot product of the rate of change and α, so that the distribution of the parameters conforms to the normal distribution. When solving the above formula, the parameters can be solved based on the idea that the maximum point is the extreme point and the partial derivative is equal to 0; it can also be solved by iterative methods, such as Gauss-Newton, gradient descent, etc.
[0074] After obtaining the coefficient vector, the coefficients of each reflectance basis function are obtained, and the spectral reflectance function can be determined by the following formula:
[0075]
[0076] Where λ represents the wavelength of light, s(λ) represents the spectral reflectance function, and b j (λ) represents the jth reflectivity basis function, α j represents the coefficient of the jth reflectivity basis function.
[0077] For the reflectivity basis function, a polynomial model may be used in some embodiments, and its expression is:
[0078]
[0079] Among them, p is the preset polynomial degree, w i,j is the coefficient of the i-th order term.
[0080] Please refer to Figure 3 , is a comparison between the reconstructed spectral reflectance and the actual value in an embodiment. It can be seen that the spectral reflectance obtained by using the spectral reflectance reconstruction method of the present invention is close to the actual value and can better reflect the imaging characteristics of the object.
[0081] The multispectral image synthesis method of the present invention is implemented based on the above-mentioned spectral reflectance reconstruction method, please refer to Figure 4 In some embodiments, the multispectral image synthesis method includes the following steps:
[0082] Step 210: Control the multi-spectral light source to project monochromatic light onto the target object, projecting M times in total, projecting monochromatic light of different wavelength bands onto the target object each time; triggering the camera to capture an image of the target object after each projection, and obtaining M N-channel images after M captures;
[0083] Step 220: Calculate the spectral reflectance function of the target object according to the spectral curve of the multi-spectral light source, the spectral response curve of the camera and the M N-channel images;
[0084] Step 230: Calculate the pixel value of each pixel point in the multispectral image of the target object using the spectral reflectance function to obtain the multispectral image of the target object.
[0085] Among them, steps 210 and 220 are the same as steps 110 and 120. In step 230, since the spectral reflectance of the object is the pixel value of the multispectral image, the spectral reflectance calculated for each pixel is different, and the reflectance of each pixel to multiple different spectral bands is calculated to form a multispectral image, so step 230 specifically includes: using the spectral reflectance function to calculate the reflectance of each pixel in the multispectral image of the target object to multiple different spectral bands, and taking the reflectance of each pixel to multiple different spectral bands as the pixel value of the pixel to obtain the multispectral image of the target object.
[0086] The multiple different spectral bands here can be obtained by linearly dividing the set bands. For example, the band with a wavelength of 400-800nm is selected in steps of 10nm to obtain bands with wavelengths of 410nm, 420nm, 430nm...
[0087] In some embodiments, before shooting, the camera needs to be calibrated for spectral response curves. The calibration method is to image N different camera channels through standard narrow-band light sources of different wavelengths, and calculate the spectral response function (SRF) according to the grayscale value. Please refer to Figure 5 In one embodiment, the spectral response curve of the camera is calibrated according to steps 310 to 340.
[0088] Step 310: Adjust multiple standard narrow-band light sources of different wavelength bands to have the same optical power.
[0089] An imaging spectrometer can be used to first detect the intensity and wavelength of the standard narrow-band light source, and then the standard narrow-band light sources of different wavelengths can be adjusted to the same optical power.
[0090] Step 320: Control the camera to respectively image a plurality of standard narrow-band light sources of different wavelength bands, and obtain a first image obtained by each channel of the camera when imaging each standard narrow-band light source.
[0091] Since the camera has N channels, N first images can be obtained when imaging each standard narrow-band light source.
[0092] Step 330: Calculate the average grayscale value of each first image.
[0093] Step 340: For the nth channel of the camera, a spectral response curve of the channel is obtained by fitting according to the average grayscale value of each first image obtained for the channel and the wavelength band of the corresponding standard narrow-band light source, wherein n=1, 2...N.
[0094] The average grayscale value of each first image and the corresponding wavelength of the standard narrow-band light source are equivalent to determining a point on the spectral response curve. The spectral response curve of the channel can be fitted according to the average grayscale value of multiple first images and the corresponding wavelength of the standard narrow-band light source.
[0095] In one embodiment, a polynomial model is used to fit the spectral response curve. The order of the polynomial model can be set according to actual needs. For example, if a fifth-order polynomial is used, the spectral response curve of the nth channel is expressed as: Where λ is the wavelength, c n,,k is the coefficient of the k-order term, c n (λ) represents the gray value. The average gray value of the first image is taken as c n (λ), the wavelength λ of the corresponding standard narrow-band light source is substituted into the polynomial model to fit the spectral response curve. The fitting method can use the least square method.
[0096] After obtaining the spectral response curves of all channels, the calibration of the camera spectral response curve is completed. Figure 6 The figure shows the spectral response curve obtained by calibrating a certain Sony CMOS camera, and from left to right are the spectral response curves of the camera's B channel, G channel, and R channel respectively.
[0097] In some embodiments, before calibrating the spectral response curve of the camera, it is necessary to calibrate the camera response function (CRF), that is, to calibrate the response of the camera gray value to the illumination brightness to linearity, so as to eliminate the influence of the camera gamma curve. Therefore, in some embodiments, before calibrating the spectral response curve of the camera, perform Figure 7 Steps 410 to 440 in .
[0098] Step 410: Control the camera to use different exposure times to image different light sources with known optical powers or the same light source to obtain multiple second images.
[0099] Step 420: Calculate the light intensity of the light source at each imaging according to the light power and the exposure time, and calculate the average grayscale value of each second image. The light intensity can be obtained by multiplying the light power by the exposure time.
[0100] Step 430: According to the average gray value of the second image and the light intensity of the corresponding light source, a curve is obtained by fitting in is the average grayscale value of the second image, and P is the ratio of the light intensity of the corresponding light source to the reference light intensity.
[0101] Similarly, the average gray value of each second image and the light intensity of the corresponding light source are equivalent to determining the curve A point on the graph can be fitted based on the average grayscale value of multiple second images and the light intensity of the corresponding light source.
[0102] Step 440: According to the curve Create a grayscale-optical power lookup table Among them I g Represents the gray value, Represents the gray value I g The corresponding optical power.
[0103] For the images subsequently captured by the camera (e.g., the images captured in steps 110, 210, and 320), according to the grayscale-optical power lookup table Get the original grayscale value of each pixel in the image value, use the corresponding The original grayscale value is replaced by the grayscale value captured by the camera, that is, the grayscale value captured by the camera is replaced by the grayscale-light power lookup table Found value.
[0104] In some embodiments, before projection, the multi-spectral light source needs to be calibrated with a spectral curve, and the calibration method is to use a standard spectrometer to detect when the multi-spectral light source projects M kinds of monochromatic lights of different wavelengths, and obtain the spectral curve of each monochromatic light. For example, for a DLP projector, a standard spectrometer is used to calculate the spectral curves for M different DLP channels. Figure 8 The figure shows the spectral curve obtained by calibration of a certain DLP projector, which are the spectral curves of the B channel, G channel and R channel from left to right.
[0105] In some embodiments, before shooting, it is necessary to correct the spatial non-uniformity of the multi-spectral light source and the camera, that is, calculate the parameters of the flat field correction, and each picture obtained in the subsequent shooting needs to be flat field corrected. Therefore, in some embodiments, before step 210, it also includes the following steps: Fig. 9 Steps 510 to 530 are shown.
[0106] Step 510: Control the multi-spectral light source to project white light onto the uniform diffuse reflection plate, trigger the camera to shoot the uniform diffuse reflection plate, and obtain a grayscale distribution map.
[0107] Step 520: Determine the global statistical value I of the grayscale value in the grayscale distribution map st Those skilled in the art will appreciate that the global statistical value I st Refers to the value obtained through global statistics, which can be the minimum value, average value, maximum value, median value, etc.
[0108] Step 530: For each pixel of the grayscale distribution map, calculate its flat field correction coefficient k x,y =I st / I x,y ; where I x,y Represents the gray value of the pixel with coordinates (x, y), k x,y Represents the flat field correction coefficient for the pixel with coordinates (x, y).
[0109] For the images subsequently captured by the camera (e.g., the images captured in steps 110 and 210), according to the formula Perform a flat field correction, where Represents the grayscale value of the pixel with coordinates (x, y) after flat field correction. That is, for each pixel of the image captured by the camera, Replace the original gray value I obtained by shooting x,y , as the gray value of the pixel.
[0110] Based on the above spectral reflectance reconstruction method, the present invention also provides a spectral reflectance reconstruction device, please refer to Fig.10 In some embodiments, the device includes a single-band image capturing module 21 and a spectral reflectance function determining module 22, which are described below respectively.
[0111] The single-band image shooting module 21 is used to control the multi-spectral light source to project monochromatic light onto the target object, a total of M times, each time projecting monochromatic light of a different band onto the target object, and triggering the camera to capture the image of the target object after each projection, and after M shots, M N-channel images are obtained, where N is the number of channels of the camera, and M and N are both integers greater than 1.
[0112] In some embodiments, the multi-spectral light source uses a DLP projector, M is the number of channels of the DLP projector, and the single-band image capture module 21 is specifically used to: control the DLP projector to project a monochrome image onto the target object, for a total of M times, each time using a different monochrome channel to project a monochrome image onto the target object.
[0113] The spectral reflectance function determination module 22 is connected to the single-band image shooting module 21, and is used to calculate the spectral reflectance function of the target object according to the spectral curve of the multi-spectral light source, the spectral response curve of the camera and M N-channel images.
[0114] In some embodiments, the spectral reflectance function is determined by a preset reflectance basis function and its coefficients. The spectral reflectance function determination module 22 calculates the spectral reflectance function of the target object based on the spectral curve of the multi-spectral light source, the spectral response curve of the camera, and M N-channel images, including: calculating the coefficients of the reflectance basis function based on the spectral curve of the multi-spectral light source, the reflectance basis function, the spectral response curve of the camera, and M N-channel images; determining the spectral reflectance function based on the reflectance basis function and the coefficients of the reflectance basis function.
[0115] In some embodiments, there are multiple reflectance basis functions, and the spectral reflectance function determination module 22 calculates the coefficients of the reflectance basis function according to the spectral curve of the multi-spectral light source, the reflectance basis function, the spectral response curve of the camera and M N-channel images, including: converting M N-channel images into M×N 1-channel images; and solving the relationship I=Fα to obtain the coefficient vector α.
[0116] Regarding the solution of the coefficient vector α in the overdetermined case, as well as how to determine the matrix F, the reflectance basis function and the spectral reflectance function, please refer to the above introduction and will not be repeated here.
[0117] Based on the above multi-spectral image synthesis method, the present invention also provides a multi-spectral image synthesis device, please refer to Fig.11 In some embodiments, the device includes a single-band image shooting module 31, a spectral reflectance function determination module 32 and a multi-spectral image synthesis module 33, wherein the single-band image shooting module 31 and the spectral reflectance function determination module 32 are the same as the above-mentioned single-band image shooting module 21 and the spectral reflectance function determination module 22, and the multi-spectral image synthesis module 33 is connected to the spectral reflectance function determination module 32, and is used to use the spectral reflectance function to calculate the reflectance of each pixel point in the multi-spectral image of the target object to multiple different spectral bands, and use the reflectance of each pixel point to multiple different spectral bands as the pixel value of the pixel point to obtain the multi-spectral image of the target object.
[0118] Please refer to Fig.12 In some embodiments, the multi-spectral image synthesis device further includes a spectral response curve calibration module 34, which is used to calibrate the spectral response curve of the camera, specifically for: adjusting multiple standard narrow-band light sources of different wavelengths to the same optical power; controlling the camera to image multiple standard narrow-band light sources of different wavelengths respectively, and obtaining the first image obtained by each channel of the camera when imaging each standard narrow-band light source; calculating the average gray value of each first image; for the nth channel of the camera, according to the average gray value of each first image obtained by the channel and the corresponding wavelength of the standard narrow-band light source, fitting the spectral response curve of the channel, wherein n = 1, 2 ... N. After obtaining the spectral response curves of all channels, the calibration of the spectral response curve of the camera is completed.
[0119] In one embodiment, a polynomial model is used to fit the spectral response curve. For details, please refer to the relevant introduction above and will not be repeated here.
[0120] Please refer to Fig.12 In some embodiments, the multi-spectral image synthesis device further includes a camera response correction module 35, which is used to: control the camera to use different exposure times to image different light sources of known light power or the same light source to obtain multiple second images before calibrating the spectral response curve of the camera; calculate the light intensity of the light source at each imaging according to the light power and the exposure time, and calculate the average gray value of each second image; and fit the curve according to the average gray value of the second image and the light intensity of the corresponding light source. in is the average gray value of the second image, and P is the ratio of the light intensity of the corresponding light source to the reference light intensity; according to the curve Create a grayscale-optical power lookup table Among them I g Represents the gray value, Represents the gray value I gFor the images subsequently captured by the camera (e.g., images captured by the single-band image capturing module 21, the single-band image capturing module 31, and the spectral response curve calibration module 34), the camera response correction module 35 calculates the corresponding optical power according to the grayscale-optical power lookup table. Get the original grayscale value of each pixel in the image value, use the corresponding The original grayscale value is replaced by the grayscale value captured by the camera, that is, the grayscale value captured by the camera is replaced by the grayscale-light power lookup table Found value.
[0121] Please refer to Fig.12 In some embodiments, the multispectral image synthesis device further includes a flat field correction module 36. The flat field correction module 36 is used to control the multispectral light source to project white light onto the uniform diffuse reflection plate, trigger the camera to shoot the uniform diffuse reflection plate, and obtain a grayscale distribution map; determine the global statistical value I of the grayscale value in the grayscale distribution map before the single-band image shooting module 31 controls the multispectral light source to project monochromatic light onto the target object. st ; For each pixel of the grayscale distribution map, calculate its flat field correction coefficient k x,y =I st / I x,y ; where I x,y Represents the gray value of the pixel with coordinates (x, y), k x,y represents the flat field correction coefficient of the pixel point with coordinates (x, y). For the images subsequently captured by the camera (e.g., images captured by the single-band image capturing module 21 and the single-band image capturing module 31), the flat field correction module 36 calculates the flat field correction coefficient according to the formula Perform a flat field correction, where Represents the grayscale value of the pixel with coordinates (x, y) after flat-field correction.
[0122] The spectral reflectance reconstruction method and device provided by the present invention obtain images of multiple different bands of the object, and calculate the spectral reflectance function of the target object according to the relationship between the spectral curve of the multi-spectral light source, the spectral reflectance function of the object, the spectral response curve of the camera and the images of different bands of the object obtained, so as to better reflect the imaging characteristics of the object. The multi-spectral image synthesis method, device and system of the present invention are implemented based on the spectral reflectance reconstruction method, and the spectral reflectance function of the target object is used to calculate the reflectance of each pixel point to multiple different spectral bands as the pixel value of the pixel point to form a multi-spectral image of the target object, so that the obtained multi-spectral image can more realistically reflect the imaging effect of the object itself. In some embodiments, the multi-spectral light source adopts a DLP projector, which has a high frame rate, high brightness, and is relatively easy to obtain, and can quickly restore the spectral reflectance information and multi-spectral image of the surface of the object at an extremely high frame rate.
[0123] Those skilled in the art will appreciate that all or part of the functions of the various methods in the above-mentioned embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, and the storage medium can include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to implement the above-mentioned functions. For example, the program is stored in the memory of the device, and when the program in the memory is executed by the processor, all or part of the above-mentioned functions can be implemented. In addition, when all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can also be stored in a storage medium such as a server, another computer, disk, optical disk, flash disk or mobile hard disk, and can be downloaded or copied and saved in the memory of the local device, or the system of the local device is updated, and when the program in the memory is executed by the processor, all or part of the functions in the above-mentioned embodiments can be implemented.
[0124] The above specific examples are used to illustrate the present invention, which is only used to help understand the present invention and is not intended to limit the present invention. For those skilled in the art, according to the concept of the present invention, some simple deductions, modifications or substitutions can be made.
Claims
1. A multispectral image synthesis method, It is characterized in that include: Control the multi-spectral light source to project monochromatic light onto the target object, projecting M times in total, projecting monochromatic light of different wavelengths onto the target object each time; After each projection, the camera is triggered to capture an image of the target object, and M N-channel images are obtained after M captures, where N is the number of channels of the camera, and both M and N are integers greater than 1; Calculating a spectral reflectance function of the target object according to a spectral curve of the multi-spectral light source, a spectral response curve of the camera, and the M N-channel images; The spectral reflectance function is used to calculate the reflectance of each pixel point in the multispectral image of the target object to multiple different spectral bands, and the reflectance of each pixel point to multiple different spectral bands is used as the pixel value of the pixel point to obtain the multispectral image of the target object.
2. The multispectral image synthesis method according to claim 1, It is characterized in that The spectral reflectance function is determined by a preset reflectance basis function and its coefficients, and the spectral reflectance function of the target object is calculated according to the spectral curve of the multi-spectral light source, the spectral response curve of the camera and the M N-channel images, including: Calculating coefficients of the reflectivity basis function according to the spectral curve of the multi-spectral light source, the reflectivity basis function, the spectral response curve of the camera, and the M N-channel images; The spectral reflectance function is determined according to the reflectance basis function and coefficients of the reflectance basis function.
3. The multispectral image synthesis method as claimed in claim 2, It is characterized in that There are multiple reflectivity basis functions, and the coefficients of the reflectivity basis functions are calculated according to the spectral curve of the multi-spectral light source, the reflectivity basis functions, the spectral response curve of the camera, and the M N-channel images, including: Convert the M N-channel images into M×N 1-channel images; The coefficients of the reflectivity basis function, the spectral curve of the multi-spectral light source, the reflectivity basis function, the spectral response curve of the camera and the M×N 1-channel images satisfy the relationship I=Fα, wherein I is a matrix composed of the M×N 1-channel images, α is a coefficient vector composed of the coefficients of all the reflectivity basis functions, and F is a matrix pre-constructed according to the spectral curve of the multi-spectral light source, the reflectivity basis function and the spectral response curve of the camera; The coefficient vector α is obtained by solving the relationship I=Fα.
4. The multispectral image synthesis method as claimed in claim 3, It is characterized in that The spectral curve of the multi-spectral light source includes spectral curves of the projected M kinds of monochromatic light, and the spectral response curve of the camera includes spectral response curves of N channels of the camera; the elements of the matrix F are determined by the following formula: F m×n,j =∫l m (l)b j (λ)c n (λ)dλ, Where, m = 1, 2 ... M, n = 1, 2 ... N, j = 1, 2 ... 8, F m×n,j represents the element in the m×nth row and jth column of the matrix F, λ represents the wavelength of light, l m (λ) represents the spectrum curve of the mth monochromatic light, c n (λ) represents the spectral response curve of the nth channel of the camera, b j (λ) represents the jth reflectivity basis function.
5. The multispectral image synthesis method as claimed in claim 3, It is characterized in that Each monochromatic light corresponds to a spectral reflectance function, each spectral reflectance function is determined by 8 reflectance basis functions, and the coefficient vector α is specifically obtained by solving the following formula: Where α* represents the coefficient vector α obtained in the final solution, b m (λ) is an M×8 matrix, each row of which is the 8 reflectance basis functions of a spectral reflectance function. Indicates b m The derivative of (λ) with respect to wavelength λ, ||·|| 2 represents the L2 norm, and γ is the set coefficient.
6. The multispectral image synthesis method according to any one of claims 2 to 5, It is characterized in that The spectral reflectance function is determined by the following formula: Where λ represents the wavelength of light, s(λ) represents the spectral reflectance function, and b j (λ) represents the jth reflectivity basis function, α j Represents the coefficient of the jth reflectivity basis function.
7. The multispectral image synthesis method according to claim 6, It is characterized in that The expression of the reflectivity basis function is: Among them, p is the preset polynomial degree, w i,j is the coefficient of the i-th order term.
8. The multispectral image synthesis method according to claim 1, It is characterized in that The spectral response curve of the camera includes the spectral response curves of N channels of the camera, and the spectral response curve of the camera is calibrated in the following manner: Adjust multiple standard narrow-band light sources of different wavelength bands to the same optical power; Controlling the camera to respectively image the plurality of standard narrow-band light sources of different wavelength bands, and acquiring a first image obtained by each channel of the camera when imaging each of the standard narrow-band light sources; Calculating the average gray value of each of the first images; For the nth channel of the camera, the spectral response curve of the channel is obtained by fitting according to the average grayscale value of each first image obtained by the channel and the wavelength band of the corresponding standard narrow-band light source, wherein n=1, 2...N.
9. The multispectral image synthesis method according to claim 8, It is characterized in that Before calibrating the spectral response curve of the camera, the method further includes: Controlling the camera to use different exposure times to image different light sources of known optical power or the same light source to obtain a plurality of second images; Calculate the light intensity of the light source during each imaging according to the light power and the exposure time, and calculate the average gray value of each of the second images; According to the average gray value of the second image and the light intensity of the corresponding light source, the curve is fitted. in is the average grayscale value of the second image, and P is the ratio of the light intensity of the corresponding light source to the reference light intensity; According to the curve Create a grayscale-optical power lookup table Among them I g Represents the gray value, Represents the gray value I g Corresponding optical power; for the image subsequently captured by the camera, according to the grayscale-optical power lookup table Get the original gray value corresponding to the image Value, use the corresponding I value to replace the original gray value.
10. The multispectral image synthesis method according to claim 1, It is characterized in that Before controlling the multi-spectral light source to project monochromatic light onto the target object, the method further includes: Controlling the multi-spectral light source to project white light onto a uniform diffuse reflection plate, triggering the camera to photograph the uniform diffuse reflection plate, and obtaining a grayscale distribution diagram; Determine the global statistical value I of the grayscale value in the grayscale distribution map st ; For each pixel of the grayscale distribution map, calculate its flat field correction coefficient k x,y =I st / I x,y ; where I x,y Represents the gray value of the pixel with coordinates (x, y), k x,y represents the flat field correction coefficient of the pixel point with coordinates (x, y); for the image subsequently taken by the camera, according to the formula Perform a flat field correction, where Represents the grayscale value of the pixel with coordinates (x, y) after flat-field correction.
11. The multispectral image synthesis method according to claim 1, It is characterized in that The multi-spectral light source is a DLP projector, and the DLP projector has M monochromatic channels of different wavelength bands; The controlling the multi-spectral light source to project monochromatic light onto the target object, projecting it M times in total, projecting monochromatic light of different bands onto the target object each time, includes: controlling the DLP projector to project a monochromatic image onto the target object, projecting it M times in total, using a different monochromatic channel to project the monochromatic image onto the target object each time.
12. A spectral reflectance reconstruction method, It is characterized in that For obtaining a spectral reflectance function of a target object, the spectral reflectance reconstruction method comprises: Control the multi-spectral light source to project monochromatic light onto the target object, projecting M times in total, projecting monochromatic light of different wavelengths onto the target object each time; triggering the camera to capture an image of the target object after each projection, and obtaining M N-channel images after M captures, where N is the number of channels of the camera, and both M and N are integers greater than 1; The spectral reflectance function of the target object is calculated according to the spectral curve of the multi-spectral light source, the spectral response curve of the camera and the M N-channel images.
13. The spectral reflectance reconstruction method according to claim 12, It is characterized in that The spectral reflectance function is determined by a preset reflectance basis function and its coefficients, and the spectral reflectance function of the target object is calculated according to the spectral curve of the multi-spectral light source, the spectral response curve of the camera and the M N-channel images, including: Calculating coefficients of the reflectivity basis function according to the spectral curve of the multi-spectral light source, the reflectivity basis function, the spectral response curve of the camera, and the M N-channel images; The spectral reflectance function is determined according to the reflectance basis function and coefficients of the reflectance basis function.
14. A multispectral image synthesis device, It is characterized in that include: A single-band image shooting module is used to control the multi-spectral light source to project monochromatic light onto the target object, projecting a total of M times, each time projecting monochromatic light of a different band onto the target object, and triggering the camera to shoot the image of the target object after each projection, and obtaining M N-channel images after M shots, where N is the number of channels of the camera, and M and N are both integers greater than 1; A spectral reflectance function determination module, connected to the single-band image shooting module, for calculating the spectral reflectance function of the target object according to the spectral curve of the multi-spectral light source, the spectral response curve of the camera and the M N-channel images; The multispectral image synthesis module is connected to the spectral reflectance function determination module, and is used to calculate the reflectance of each pixel point in the multispectral image of the target object to multiple different spectral bands using the spectral reflectance function, and use the reflectance of each pixel point to multiple different spectral bands as the pixel value of the pixel point to obtain the multispectral image of the target object.
15. The multi-spectral image synthesis device according to claim 14, It is characterized in that The spectral reflectance function is determined by a preset reflectance basis function and its coefficients. The spectral reflectance function determination module calculates the spectral reflectance function of the target object according to the spectral curve of the multi-spectral light source, the spectral response curve of the camera and the M N-channel images, including: Calculating coefficients of the reflectivity basis function according to the spectral curve of the multi-spectral light source, the reflectivity basis function, the spectral response curve of the camera, and the M N-channel images; The spectral reflectance function is determined according to the reflectance basis function and coefficients of the reflectance basis function.
16. A spectral reflectance reconstruction device, It is characterized in that For obtaining a spectral reflectance function of a target object, the spectral reflectance reconstruction device comprises: A single-band image shooting module is used to control the multi-spectral light source to project monochromatic light onto the target object, projecting a total of M times, each time projecting monochromatic light of a different band onto the target object, and triggering the camera to shoot the image of the target object after each projection, and obtaining M N-channel images after M shots, where N is the number of channels of the camera, and M and N are both integers greater than 1; The spectral reflectance function determination module is connected to the single-band image shooting module and is used to calculate the spectral reflectance function of the target object according to the spectral curve of the multi-spectral light source, the spectral response curve of the camera and the M N-channel images.
17. A multispectral image synthesis system, It is characterized in that include: Multi-spectral light source, used to project monochromatic light in different wavelength bands; A camera, used to capture images of the target object; A display, used to display the multispectral image of the target object; A controller, connected to the multi-spectral light source, the camera and the display, is configured to perform the following steps: Control the multi-spectral light source to project monochromatic light onto the target object M times in total, projecting monochromatic light of different wavelength bands onto the target object each time; trigger the camera to capture an image of the target object after each projection, and obtain M N-channel images after M captures, where N is the number of channels of the camera, and both M and N are integers greater than 1; Calculating a spectral reflectance function of the target object according to a spectral curve of the multi-spectral light source, a spectral response curve of the camera, and the M N-channel images; The spectral reflectance function is used to calculate the reflectance of each pixel point in the multispectral image of the target object to multiple different spectral bands, and the reflectance of each pixel point to multiple different spectral bands is used as the pixel value of the pixel point to obtain the multispectral image of the target object; The multispectral image of the target object is output to the display for display.
18. A computer-readable storage medium, It is characterized in that The medium stores a program, which can be executed by a processor to implement the multispectral image synthesis method according to any one of claims 1 to 11 or the spectral reflectance reconstruction method according to any one of claims 12 to 13.
Citation Information
Cited By
UHPC decorative plate pressing quality detection method based on vision
CN120385683A
Screen brightness detection method and system based on spectral waveform
CN121231022A
