Prism split multispectral camera sensor image plane matching method and device

Abstract

The application discloses a prism split light multispectral camera sensor image plane matching method and device. The application uses a projection device to irradiate a sensor through a multi-split prism and reflect to a camera to form phase deflection, and three-dimensional reconstruction is performed on the surface information of the sensor. The projection device contains multiple waveband light sources, and through the multi-split prism, three-dimensional information can be formed on the sensor at different exit surface positions. Automatic adjustment is performed on the prism bonding support by using an electric translation stage and an electric angle position stage, the spatial position of the sensor can be adjusted in real time during the prism bonding process, and multi-image sensor image plane matching calibration is realized. The application adopts phase deflection to perform three-dimensional reconstruction on the surface information of the sensor, avoids the influence of lateral chromatic aberration and distortion, and ensures accurate matching when multiple sensors are bonded.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of prism spectrometer camera imaging technology, in particular to a prism spectrometer camera sensor image plane matching method and device. BACKGROUND

[0002] The prism spectrometer camera adopts dichroic film layer in the multi-prism to divide the light of different wavebands into multiple independent black and white sensors in space for simultaneous imaging of multi-spectral precise spatial alignment. Position deviation occurs during the bonding process of the sensor and the prism, causing image difference and spatial position misalignment, affecting the image matching degree between different sensors. The present application proposes a method of using phase deflection to reconstruct the three-dimensional information of the sensor surface to assist in the matching and calibration of multiple image sensors, achieving precise matching of multiple image sensors in spatial position offset and angle deviation, and ensuring the spatial position consistency and imaging clarity consistency of each pixel point information on the spectral image.

[0003] The prism spectrometer camera needs to synthesize images taken under different spectra, requiring spatial alignment of images under different spectra. Therefore, during the bonding process of the sensor and the prism, the spatial position of the sensor needs to be adjusted on the transmission and reflection surfaces of the multi-prism to ensure the spatial and imaging clarity consistency of different spectral images and reduce the algorithm burden of the rear-end image fusion. The existing multi-image sensor registration method requires the introduction of a lens and a target, and the sensor captures the pattern on the target with the help of the lens, and adjusts the spatial installation position of the sensor according to the clarity difference at different positions.

[0004] In summary, the existing multi-image sensor image plane matching and calibration method has the following shortcomings:

[0005] (1) The angle rotation of X-axis and Y-axis needs to be adjusted manually, which lacks automation and has low precision in manual adjustment.

[0006] (2) The ordinary objective lens itself has lateral chromatic aberration and distortion, which will affect the adjustment of X-axis, Y-axis and three angles. The image captured by the objective lens is not accurate enough to determine the deviation of X, Y and angle, which may cause rework and low efficiency. SUMMARY

[0007] The prism spectrometer camera sensor image plane matching method and device proposed by the present application can at least solve one of the above technical problems.

[0008] To achieve the above purpose, the present application proposes the following technical solutions:

[0009] A prism split multispectral camera sensor image plane matching method, the multispectral camera uses a multi-split prism to split light of different wavebands to a plurality of corresponding image sensors, the method is used for matching the positional relationship of the plurality of image sensors, so that the spatial position and clarity of the object information obtained by the plurality of image sensors are consistent, and the method comprises:

[0010] A preset reference image sensor is fixed on a corresponding position of the multi-split prism, and an image sensor to be matched is pre-fixed on a corresponding position of the multi-split prism;

[0011] Different wavebands of light are used to project grating fringes through the multi-split prism to the corresponding image sensors, the camera collects grating fringe images reflected by each image sensor, and three-dimensional reconstruction is performed on the surface of each image sensor according to the grating fringe images collected by the camera using phase deflection technology;

[0012] Position deviation between each image sensor to be matched and the reference image sensor is obtained according to the three-dimensional reconstruction result, and the position of each image sensor to be matched is initially adjusted according to the deviation;

[0013] The multispectral camera is provided with a telecentric lens to collect a resolution test target, position deviation between each image sensor to be matched and the reference image sensor is obtained according to the resolution test target collected by each image sensor, and the position of each image sensor to be matched is finely adjusted according to the deviation.

[0014] Further, the different wavebands of light are used to project grating fringes through the multi-split prism to the corresponding image sensors, and the camera collects grating fringes reflected by each image sensor, comprising:

[0015] Grating fringes with horizontal and vertical intersections are projected into the multi-split prism using light of a waveband corresponding to the reference image sensor, and the grating fringes reach the reference image sensor after being split by the multi-split prism; the grating fringes reflected by the reference image sensor are collected by the camera after passing through the multi-split prism, and a grating fringe image is obtained;

[0016] The above steps are performed using light of a waveband corresponding to each image sensor respectively, and a grating fringe image reflected by each image sensor is obtained.

[0017] Further, the three-dimensional reconstruction of the surface of each image sensor is performed using phase deflection technology according to the grating fringe images collected by the camera, comprising:

[0018] Fourier transform is performed on the horizontal and vertical directions of the grating fringe image respectively;

[0019] The base frequency signals are obtained after filtering in the horizontal and vertical directions respectively, and Fourier transform is performed again, so that phase images in the horizontal and vertical directions are obtained;

[0020] The three-dimensional phase field in the phase diagram is phase-unfolded along the time axis direction for each pixel respectively, and a three-dimensional surface shape of the sensor surface is obtained through a height reconstruction algorithm based on radial basis function integration according to the known camera internal parameter and the projection light source position information of the grating fringe, so as to complete the three-dimensional reconstruction.

[0021] Further, the position deviation between each to-be-matched image sensor and the reference image sensor is obtained according to the three-dimensional reconstruction result, and the position of each to-be-matched image sensor is initially adjusted according to the deviation, including:

[0022] The three-dimensional surface shape of each to-be-matched image sensor is matched with the three-dimensional surface shape of the reference image sensor to obtain a corresponding matching feature point set;

[0023] The rotation vector and the translation vector between the feature point set of each to-be-matched image sensor and the corresponding feature point set of the reference image sensor are calculated to obtain the position deviation, including the deviation X1, Y1 and Z1 of the X, Y and Z axes and the angle deviation θ(X1), θ(Y1) and θ(Z1) of the X, Y and Z directions between each to-be-matched image sensor and the reference image sensor;

[0024] The position of each image sensor is fine-adjusted according to the position deviation.

[0025] Further, the multispectral camera is equipped with a telecentric lens to collect a resolution test target, the position deviation between each to-be-matched image sensor and the reference image sensor is obtained according to the resolution test target collected by each image sensor, and the position of each to-be-matched image sensor is fine-adjusted according to the deviation, including:

[0026] The multispectral camera is equipped with a telecentric lens to collect the same resolution test target for multiple times;

[0027] For the resolution test target collected by each image sensor each time, the following steps are performed:

[0028] The resolution test target collected by each to-be-matched image sensor is matched with the resolution test target collected by the reference image sensor in terms of feature points;

[0029] The rotation vector and the translation vector between the feature point set of the resolution test target collected by each to-be-matched image sensor and the feature point set of the resolution test target collected by the corresponding reference image sensor are calculated to obtain the position deviation, including the offset X2 and Y2 of the X and Y axes and the rotation angle θ(Z2) of the Z direction;

[0030] Obtaining the average value of the X, Y axis offset X2, Y2 and the rotation angle θ(Z2) in the Z direction of each image sensor after collecting the resolution test target image each time, and obtaining the average position deviation;

[0031] Fine-tuning the position of each image sensor to be matched according to the average position deviation.

[0032] Further, the fine-tuning of the position of each image sensor to be matched according to the average position deviation comprises:

[0033] Judging whether the average value of each offset of the X, Y axis offset X2, Y2 and the rotation angle θ(Z2) in the Z direction is greater than the set compensation threshold value, and if the average value of any offset is less than the corresponding compensation threshold value, the offset is not fine-tuned;

[0034] If the average value of any offset is not less than the corresponding compensation threshold value, the offset is fine-tuned, and the steps of collecting the same resolution test target multiple times by the multispectral camera with the telecentric lens and calculating the average position deviation are repeated until the average value of each offset of the X, Y axis offset X2, Y2 and the rotation angle θ(Z2) in the Z direction is less than the set compensation threshold value.

[0035] On the other hand, the present application also proposes a prism spectrometer multispectral camera sensor image plane matching device, which uses a multi-split prism to split light of different wavebands onto corresponding image sensors, and the method is used for matching the position relationship of the image sensors to make the spatial position and clarity of the object information obtained by the image sensors consistent, comprising:

[0036] A six-degree-of-freedom adjustment table is used to adjust the position of the image sensor to be matched, and the image sensor to be matched is pre-fixed on the corresponding position of the multi-split prism, and the reference image sensor is fixed on the multi-split prism;

[0037] An XYZ scanning detection table is used to adjust the position of the projection device and the camera, the telecentric lens and the resolution test target;

[0038] The XYZ scanning detection table adjusts the position of the projection device and the camera, so that the projection device projects light of different wavebands through the multi-split prism onto the corresponding image sensor, the camera collects the grating fringe reflected by each image sensor, the surface of each image sensor is three-dimensionally reconstructed by using phase deflection according to the grating fringe collected by the camera, and the position deviation between each image sensor to be matched and the reference image sensor is obtained according to the three-dimensional reconstruction result; and the six-degree-of-freedom adjustment table preliminarily adjusts the position of each image sensor to be matched according to the deviation;

[0039] The XYZ scanning detection table adjusts the position of the telecentric lens and the resolution test target, so that the resolution test target projected by the resolution test target passes through the telecentric lens to the multi-splitting prism, and is collected by each image sensor; the position deviation between each to-be-matched image sensor and the reference image sensor is obtained according to the plurality of resolution test target images collected by each image sensor; and the six-degree-of-freedom adjustment table fine-tunes the position of each to-be-matched image sensor according to the deviation.

[0040] Further, the six-degree-of-freedom adjustment table comprises:

[0041] An XYZ axis translation table is configured to adjust the deviation of the X, Y and Z axes between the to-be-matched image sensor and the reference image sensor.

[0042] An XYZ axis angle table is configured to adjust the angle deviation θ(X1), θ(Y1) and θ(Z1) of the X, Y and Z directions between the to-be-matched image sensor and the reference image sensor.

[0043] Further, the XYZ scanning detection table comprises:

[0044] An XYZ axis displacement platform is connected with the adapter plate.

[0045] The adapter plate is installed with the projection device and the camera, the telecentric lens and the resolution test target.

[0046] The XYZ axis displacement platform is configured to adjust the position of the projection device and the camera, the telecentric lens and the resolution test target on the X, Y and Z axes.

[0047] Further, the six-degree-of-freedom adjustment table comprises:

[0048] A prism bonding structure comprises a prism fixing frame and a sensor bonding plate, wherein the prism fixing frame is configured to fix the multi-splitting prism, and the sensor bonding plate is fixed with the image sensor on one side and connected with the prism fixing frame on the other side.

[0049] The bonding plate fixed with the reference image sensor on one side is directly fixed at the corresponding position of the prism fixing frame, so that the multi-splitting prism irradiates the received light source onto the reference image sensor after splitting the light source.

[0050] The bonding plate fixed with the to-be-matched image sensor on one side is pre-fixed at the corresponding position of the prism fixing frame, so that the multi-splitting prism irradiates the received light source onto the reference image sensor after splitting the light source.

[0051] The other side of the bonding plate fixed with the to-be-matched image sensor is connected with the six-degree-of-freedom adjustment table, and the position of the to-be-matched image sensor is adjusted under the driving of the six-degree-of-freedom adjustment table.

[0052] After the position adjustment of the image sensor to be matched is completed, the bonding plate with the image sensor to be matched fixed on one side is fixed on the prism fixing frame.

[0053] The beneficial effects of the present application are as follows:

[0054] (1) The present application adopts phase deflection to perform three-dimensional reconstruction on the surface information of the sensor, avoids the influence of lateral chromatic aberration and distortion, and ensures the accurate matching when multiple sensors are bonded.

[0055] (2) The present application can achieve sub-pixel level matching accuracy through the initial adjustment and fine adjustment of the position of the image sensor to be matched, and has higher accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 is a schematic diagram of the prism splitting multispectral camera sensor image plane matching device of the present application;

[0057] Figure 2 is a schematic diagram of the six-degree-of-freedom adjustment table in the embodiment of the present application;

[0058] Figure 3 is a schematic diagram of the XYZ scanning detection table in the embodiment of the present application;

[0059] Figure 4 is a schematic diagram of the prism bonding structure in the embodiment of the present application;

[0060] Figure 5 is a schematic diagram of the prism fixing frame in the embodiment of the present application;

[0061] Figure 6 is a schematic diagram of the projection device and camera acquisition grating fringe image process in the embodiment of the present application;

[0062] Figure 7 is a three-dimensional reconstruction flowchart of the surface information of the image sensor in the embodiment of the present application;

[0063] Figure 8 is a schematic diagram of slope calculation on the grating fringe image in the embodiment of the present application;

[0064] Figure 9 is a schematic diagram of the position of the reference image sensor irradiated by the ultraviolet lamp in the embodiment of the present application;

[0065] Figure 10 is a schematic diagram of the position of the image sensor to be matched irradiated by the ultraviolet lamp in the embodiment of the present application;

[0066] Figure 11 is an operation flowchart of the prism splitting multispectral camera sensor image plane matching device in the embodiment of the present application.

[0067] In the figure: 1-air floating foot pad; 2-marble base; 3-mounting platform; 4-XYZ axis translation stage; 5-XYZ axis angle position stage; 6-XYZ scanning detection stage; 7-projection device; 8-camera; 9-telecentric lens; 10-resolution test target; 11-prism bonding structure; 11a-prism fixing frame; 11b-sensor bonding plate; 12-six-degree-of-freedom adjustment stage. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments.

[0069] The embodiment first proposes a prism spectrometer camera sensor image plane matching device, which is applied to image sensor image plane matching of a two-spectrometer camera. It can be understood that when the two-spectrometer prism of the two-spectrometer camera is expanded to a multi-spectrometer prism, only a simple improvement of the device can be obtained by those skilled in the art without creative thinking, that is, the image plane matching of multiple image sensors of a multi-spectrometer camera can be realized.

[0070] The prism spectrometer camera sensor image plane matching device of the embodiment is shown in Figure 1 The base body is used for carrying other devices. Because there is inevitable jitter in the laboratory environment, image offset is easily generated during image acquisition. The marble base 2 has high stability, and the air floating foot pad 1 has good vibration isolation capability. The combination of the two can greatly reduce the environmental impact.

[0071] The mounting platform 3 is respectively provided with the six-degree-of-freedom adjustment stage 12 and the XYZ scanning detection stage 6. The six-degree-of-freedom adjustment stage 12 is shown in Figure 2 The six-degree-of-freedom adjustment stage 12 includes the XYZ axis translation stage 4 and the XYZ axis angle position stage 5. The XYZ axis translation stage 4 is used to provide translation on the X, Y and Z axes, and the XYZ axis angle position stage 5 is used to provide rotation on the X, Y and Z axes. The XYZ scanning detection stage 6 is provided with the projection device 7, the camera 8, the telecentric lens 9 and the resolution test target 10. Figure 3As shown, the XYZ scanning detection table 6 is used to adjust the positions of the projection device 7 and the camera 8, the telecentric lens 9 and the resolution test target 10 in the X, Y and Z axes. The XYZ scanning detection table 6 can adjust the positions of the projection device 7 and the camera 8 so that the projection device 7 projects the grating fringes through the multi-split prism to the corresponding image sensor, and then the camera 8 collects the grating fringe images reflected by each image sensor; the XYZ scanning detection table 6 can also move the projection device 7 and the camera 8 away, and then adjust the positions of the telecentric lens 9 and the resolution test target 10 so that the resolution test target images projected by the resolution test target 10 pass through the telecentric lens 9 to the multi-split prism and are collected by each image sensor, wherein the resolution test target has a plurality of sets of line pairs with equal line spacing and line width, and the USAF 1951 resolution test target is preferably used in the embodiment.

[0072] The prism bonding structure 11 is also mounted on the mounting platform 3, as shown. Figure 4 The prism bonding structure 11 includes a prism fixing frame 11a and a sensor bonding plate 11b, as shown. Figure 5 The sensor bonding plate 11b has an image sensor fixed on one side and is connected to the prism fixing frame 11a on the other side. In the embodiment, two image sensors need to be matched, one of which is a reference image sensor and the other is a to-be-matched image sensor. The two sensor bonding plates 11b respectively mount the two image sensors at corresponding positions of the multi-split prism, which allows the multi-split prism to irradiate different wavebands of light onto the reference image sensor and the to-be-matched image sensor after splitting the received light source. In the matching process, the bonding plate with the reference image sensor fixed on one side is directly fixed at the corresponding position of the prism fixing frame 11a, and the bonding plate with the to-be-matched image sensor fixed on one side is first pre-fixed at the corresponding position of the prism fixing frame 11a, which means that the to-be-matched image sensor can be temporarily fixed on the prism fixing frame 11a but can change its pre-fixed position under external force. The other side is connected to the six-degree-of-freedom adjustment table 12, which adjusts the position of the to-be-matched image sensor under the driving of the six-degree-of-freedom adjustment table 12. After the position adjustment of the to-be-matched image sensor is completed, the bonding plate with the to-be-matched image sensor fixed on one side is fixed on the prism fixing frame 11a.

[0073] The prism splitting multi-spectral camera sensor image plane matching method of the embodiment includes the following steps:

[0074] Step 100, installation of the two-split prism and the image sensor, including:

[0075] Step 110, installation of the two-split prism into the prism fixing frame 11a, and installation of the prism fixing frame 11a into the mounting platform 3.

[0076] Step 120, the reference image sensor and the image sensor to be matched are connected with the bonding plate, the bonding plate with the reference image sensor is directly fixed on the prism fixing frame 11a, and the bonding plate with the image sensor to be matched is pre-fixed on the prism frame and connected with the six-degree-of-freedom adjusting table 12 at one end.

[0077] Step 130, the XY axis of the XYZ scanning detection platform is moved to ensure that the projection device 7 can be projected onto the sensor through the two-split prism and the camera 8 can collect the image reflected by the image sensor; then the Z axis of the XYZ scanning detection platform is moved to ensure that the image collected by the reference image sensor is clear, and then the Z axis of the six-degree-of-freedom adjusting table 12 is moved to ensure that the image collected by the image sensor to be matched is clear; wherein the positions of the XY and the two Z axes are manually determined at the first test, and the positions are recorded at this time, and subsequent tests can be directly programmed to move to the corresponding positions without manual determination again.

[0078] Step 200, three-dimensional reconstruction of the sensor surface is performed by using phase deflection, and the position of the image sensor to be matched is initially adjusted, which specifically includes:

[0079] Step 210, system calibration: the camera 8 and the projection device 7 are calibrated, including the calibration of the camera 8 internal parameter and the spatial position calibration of the projection device 7, which is performed at the first operation and does not need to be repeated after the calibration result is saved.

[0080] Step 220, the projection device 7 includes two light sources of spectral band A and spectral band B; first, the spectral band A of the light source of the projection device 7 is turned on, the projection device 7 projects the grating fringe onto the two-split prism, the dichroic film layer divides the spectral band A to the reference image sensor, and the camera 8 collects the grating fringe image reflected by the reference image sensor through the two-split prism, as shown in Figure 6 .

[0081] Step 230, phase extraction is performed according to the collected cross grating fringe. The extraction method adopted in this embodiment is Fourier transform profilometry, the projection device 7 projects the cross fringe, the camera 8 collects the fringe reflected by the reference image sensor, Fourier transform is performed on the X and Y directions respectively, the fundamental frequency signal is filtered and inverse Fourier transform is performed, and the phase image is calculated, as shown in Figure 7 .

[0082] Step 240, the phase extracted from the deformed grating fringe pattern by the phase extraction algorithm is called truncated phase, which needs to be unwrapped into continuous phase by phase unwrapping algorithm. The time phase unwrapping method is adopted in the present application, which uses the three-dimensional phase field obtained from multiple sets of grating fringe patterns with different frequencies to unwrap the phase of each pixel along the time axis direction one by one, and then calculates the slope with the camera 8 internal parameter and the calibration result of the spatial position of the projection device 7, which is the gradient of the sensor surface, as shown in Figure 8 .

[0083] Step 250, according to the obtained slope information, the three-dimensional surface shape of the reference image sensor can be obtained by height reconstruction algorithm. The radial basis function integral reconstruction method is adopted in the present application, which has high reconstruction accuracy for small deformation areas and can well handle irregular data.

[0084] Step 260, after obtaining the three-dimensional surface shape of the reference image sensor, switch the spectral band of the light source of the projection device 7 to spectral band B, and repeat steps 230, 240 and 250 to obtain the three-dimensional surface shape of the image sensor to be matched.

[0085] Step 270, taking the reference image sensor as the reference target, the spatial position of the image sensor to be matched is matched and calibrated, which specifically includes:

[0086] Step 271, feature point matching is performed on the surface shapes of the reference image sensor and the image sensor to be matched to form matching feature point sets {Pr1} and {Pt1}, and the rotation and translation vectors between the two point sets are calculated to obtain the deviations X1, Y1 and Z1 of XYZ axes and the angle deviations θ(X1), θ(Y1) and θ(Z1) of XYZ directions.

[0087] Step 272, the six-degree-of-freedom adjustment table 12 automatically adjusts the spatial position of the image sensor to be matched according to the obtained rotation and translation parameters.

[0088] Since the sampling of the camera 8 on the measured object is discrete, the information in the adjacent two pixels of the measured surface shape gradient cannot be completely reconstructed, so the original surface shape cannot be completely and accurately restored. The longitudinal Z matching accuracy of the three-dimensional surface shape reconstruction and matching method can reach within the focal depth of the lens, but the lateral XY matching accuracy can only reach the pixel level. To completely match the images collected by multiple image sensors, the lateral matching accuracy must be sub-pixel level. Therefore, the adjustment of the above steps cannot meet the final requirements. A telecentric lens 9 is needed to be added and matched with a resolution test target 10, and further adjustment is needed. The chromatic aberration and distortion of the ordinary lens itself will inevitably affect the XY direction calibration. The telecentric lens 9 has low distortion and constant magnification, which can eliminate the influence of distortion and lateral chromatic aberration.

[0089] The embodiment also includes the following steps:

[0090] Step 300, the position of the image sensor to be matched is initially adjusted by using the telecentric lens 9 and matching the resolution test target 10, and the specific steps are as follows:

[0091] Step 310, the XY axes of the XYZ scanning detection platform are moved to adjust the positions of the telecentric lens 9 and the resolution test target 10, so that the resolution test target image projected by the resolution test target 10 passes through the telecentric lens 9 to the multi-split prism and is collected by each image sensor. The Z axis of the XYZ scanning detection platform is moved to adjust the image collected by the reference image sensor to be clear, wherein the position of the telecentric lens 9 and the position of the Z axis when the image of the reference image sensor is clear are recorded at the first test, and the corresponding positions can be directly moved according to the saved position information at the subsequent test.

[0092] Step 320, the reference image sensor and the image sensor to be matched collect the resolution test target image. In order to improve the accuracy, the embodiment continuously collects multiple images at the same time, and one preferred embodiment is to collect 50 images. The selected image center contains the region with alternating bright and dark stripes. Feature point matching is performed on the two images collected each time to form the matching feature point sets {Pr2} and {Pt2}, and the rotation and translation vectors between the two point sets are calculated. The rotation and translation amounts obtained by multiple imaging are averaged to obtain the offset amounts X2, Y2 of the XY axes and the rotation angle θ(Z2) in the Z direction. Then, the relationship between X2, Y2 and θ(Z2) and the corresponding compensation threshold is judged. When any of the three parameters is not less than the set compensation threshold, the X-axis, Y-axis translation stages and Z-axis angle stages of the six-degree-of-freedom adjustment table 12 are finely adjusted according to the corresponding offset amount and rotation angle. The operation of this step is repeated until all three parameters are less than the set compensation threshold.

[0093] Step 400, the image sensor to be matched is fixed on the prism frame, and the prism split multi-spectral camera sensor image plane matching is completed.

[0094] In one embodiment, the image sensor and the prism frame are fixed by ultraviolet curing glue, Figure 9 and Figure 10 respectively represent the irradiation positions of the ultraviolet lamp when the reference image sensor and the image sensor to be matched are fixed by ultraviolet curing glue. In addition, each component in the embodiment is automatically controlled by a program, and the specific control process is shown in Figure 11 as follows:

[0095] (1) The reference image sensor is fixed with the to-be-matched image sensor and the respective sensor bonding plate 11b, the reference image sensor bonding plate 11b is fixedly bonded with the prism frame, and the to-be-matched image sensor bonding plate 11b is bonded with the prism frame with ultraviolet curing glue, but ultraviolet lamp irradiation is not performed at present.

[0096] (2) The XY axis of the XYZ scanning detection platform is moved through a program, to ensure that the projection device 7 can be projected onto the image sensor through the two-split prism, and the camera 8 can collect the image reflected back by the sensor; then the Z axis of the XYZ scanning detection platform is moved through a program, to ensure that the image collected by the reference image sensor is clear, and then the Z axis of the six-degree-of-freedom adjustment table 12 is moved through a program, to ensure that the image collected by the to-be-matched image sensor is clear; wherein the positions of the XY axis and the two Z axes are manually determined at the first time of testing, and the positions are recorded, and subsequent testing can be directly moved to the corresponding positions through a program without manual determination again.

[0097] (3) The surface information of the reference image sensor and the to-be-matched image sensor is reconstructed in three dimensions, and the adjustment parameters are calculated, the program inputs the feedback parameters into the six-degree-of-freedom adjustment table 12, and the initial adjustment of the spatial position of the to-be-matched image sensor is performed.

[0098] (4) The XY axis of the XYZ scanning detection platform is moved, the telecentric lens 9 and the resolution test target 10 are switched, the Z axis of the XYZ scanning detection platform is moved to adjust the image collected by the reference image sensor to be clear, and the second spatial position adjustment of the to-be-matched image sensor is performed, the XY offset and the rotation angle in the Z direction are adjusted through the six-degree-of-freedom adjustment table 12. At this point, the spatial position of the to-be-matched image sensor before curing has been adjusted; wherein the position of the telecentric lens 9 and the position of the Z axis when the image of the to-be-matched image sensor is clear can be recorded at the first time of testing, and subsequent testing can be directly moved to the corresponding positions through a program.

[0099] (5) Ultraviolet lamp curing, after the spatial position adjustment of the to-be-matched image sensor is completed, the program controls the ultraviolet lamp to be turned on, and the ultraviolet lamp is automatically turned off after 30 minutes, and the bonding of the to-be-matched image sensor and the prism is completed after the end.

[0100] The entire process of the embodiment is recorded after the first operation, and subsequent automatic operation can be performed without manual operation.

[0101] The technical features of the above embodiments can be combined arbitrarily, and to make the description concise, all possible combinations of the technical features in the above embodiments are not described, but as long as the combinations of the technical features do not exist contradictory, they should be considered as the scope of the description.

[0102] The above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can be modified, or some technical features can be replaced by equivalent features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A prism spectrometer camera sensor image plane matching method, the prism spectrometer camera uses a multi-prism to divide light of different wavebands to a plurality of corresponding image sensors, the method is used for matching the positional relationship of the plurality of image sensors, so that the spatial position and clarity of the object information obtained by the plurality of image sensors are consistent, characterized in that, include: ​ Fix the preset reference image sensor at the corresponding position on the multi-prism; The image sensor to be matched is pre-fixed at the corresponding position of the multi-prism; Different wavelengths of light are used to project grating fringes onto the corresponding image sensors through a multi-prism. The camera captures the grating fringe pattern reflected from each image sensor. Based on the grating fringe pattern captured by the camera, phase deflection is used to perform three-dimensional reconstruction of the surface of each image sensor. Based on the 3D reconstruction results, the positional deviation between each image sensor to be matched and the reference image sensor is obtained, and the position of each image sensor to be matched is initially adjusted based on the deviation. The multispectral camera, equipped with a telecentric lens, acquires resolution test target images. Based on the resolution test target images acquired by each image sensor, the positional deviation between each image sensor to be matched and the reference image sensor is obtained. The position of each image sensor to be matched is then finely adjusted based on the deviation.

2. The prism-based multispectral camera sensor image plane matching method according to claim 1, characterized in that, The process involves projecting grating fringes onto corresponding image sensors using light of different wavelengths through a multi-prism, with the camera capturing the grating fringes reflected from each image sensor, including: Using light in the wavelength band corresponding to the reference image sensor, crisscrossing grating stripes are projected onto a multi-prism. After being split by the multi-prism, the light reaches the reference image sensor. The grating stripes reflected by the reference image sensor are then captured by the camera after passing through the multi-prism to obtain a grating stripe pattern. The above steps are performed using light in the corresponding wavelength bands of each of the other image sensors to obtain the grating fringe pattern reflected by each image sensor.

3. The prism-based multispectral camera sensor image plane matching method according to claim 2, characterized in that, The step of using phase deflection to perform three-dimensional reconstruction of each image sensor surface based on the grating fringe pattern acquired by the camera includes: Perform Fourier transforms on the horizontal and vertical axes of the grating fringe pattern, respectively; After filtering in the horizontal and vertical directions respectively, the fundamental frequency signal is obtained, and then Fourier transform is performed to obtain the phase diagrams in the horizontal and vertical directions. The three-dimensional phase field in the phase map is unfolded for each pixel along the time axis. Then, based on the known camera intrinsic parameters and the position information of the projection light source of the grating fringes, the three-dimensional surface shape of the sensor is obtained through a height reconstruction algorithm based on radial basis function integration, thus completing the three-dimensional reconstruction.

4. The prism-based multispectral camera sensor image plane matching method according to claim 3, characterized in that, The step of obtaining the positional deviation between each image sensor to be matched and the reference image sensor based on the 3D reconstruction results, and initially adjusting the position of each image sensor to be matched based on the deviation, includes: The 3D surface shape of each image sensor to be matched is matched with the 3D surface shape of the reference image sensor to obtain the corresponding matching feature point set; Calculate the rotation vector and translation vector between the feature point set of each image sensor to be matched and the feature point set of the corresponding reference image sensor, and obtain the positional deviation, including the deviations X1, Y1, Z1 of each image sensor to be matched and the reference image sensor along the X, Y, and Z axes, and the angular offsets θ(X1), θ(Y1), θ(Z1) in the X, Y, and Z directions. The position of each image sensor is initially adjusted based on the positional deviation.

5. The prism-based multispectral camera sensor image plane matching method according to claim 1, characterized in that, The multispectral camera, equipped with a telecentric lens, acquires resolution test target images. Based on the resolution test target images acquired by each image sensor, the positional deviation between each image sensor to be matched and the reference image sensor is obtained. The position of each image sensor to be matched is then fine-tuned based on these deviations, including: A multispectral camera equipped with a telecentric lens repeatedly acquires test target images at the same resolution. For each resolution test target image acquired by each image sensor, perform the following steps: For each target image image to be matched, the resolution test target image acquired by the image sensor to be matched is matched with the resolution test target image acquired by the reference image sensor. Calculate the rotation and translation vectors between the feature point set of the resolution test target image acquired by each image sensor to be matched and the feature point set of the corresponding reference image sensor acquired by the resolution test target image, and obtain the positional deviation, including the offsets X2 and Y2 of the X and Y axes and the rotation angle θ (Z2) in the Z direction. The average position deviation is obtained by calculating the average values ​​of the offsets X2 and Y2 along the X and Y axes and the rotation angle θ (Z2) in the Z direction after each image sensor acquires a target image for resolution testing. The position of each image sensor to be matched is fine-tuned based on the average positional deviation.

6. The prism-based multispectral camera sensor image plane matching method according to claim 5, characterized in that, The step of fine-tuning the position of each image sensor to be matched based on the average position deviation includes: Determine whether the average value of each offset among the offsets X2 and Y2 of the X and Y axes and the rotation angle θ (Z2) in the Z direction is greater than a set compensation threshold. If the average value of any offset is less than the corresponding compensation threshold, then the offset is not fine-tuned. If the average value of any offset is not less than the corresponding compensation threshold, the offset is fine-tuned. After fine-tuning, the steps of acquiring test target images of the same resolution multiple times with the multispectral camera equipped with a telecentric lens and calculating the average position deviation are repeated until the average value of each offset in the X and Y axis offsets X2 and Y2 and the rotation angle θ (Z2) in the Z direction is less than the set compensation threshold.

7. A prism-based multispectral camera sensor image plane matching device, used to implement the prism-based multispectral camera sensor image plane matching method as described in any one of claims 1-6, wherein the multispectral camera uses a multi-splitting prism to split light of different wavelengths onto multiple corresponding image sensors, and the method is used to match the positional relationship of the multiple image sensors so that the spatial position and sharpness of the object information acquired by the multiple image sensors are consistent, characterized in that... include: A six-degree-of-freedom adjustment stage is used to adjust the position of the image sensor to be matched. The image sensor to be matched is pre-fixed at the corresponding position of the multi-beam prism, and the reference image sensor is fixed on the multi-beam prism. The XYZ scanning inspection stage is used to adjust the position of the projection device and camera, telecentric lens and resolution test target; The XYZ scanning detection stage adjusts the positions of the projection device and camera, allowing the projection device to project light of different wavelengths onto the corresponding image sensors via a multi-prism. The camera captures the grating fringe pattern reflected from each image sensor. Based on the grating fringe pattern captured by the camera, phase deflection is used to perform three-dimensional reconstruction of the surface of each image sensor. The positional deviation between each image sensor to be matched and the reference image sensor is obtained based on the three-dimensional reconstruction results. The six-degree-of-freedom adjustment stage initially adjusts the position of each image sensor to be matched based on the deviation. The XYZ scanning detection stage adjusts the position of the telecentric lens and the resolution test target so that the resolution test target image projected by the resolution test target passes through the telecentric lens to the multi-beam prism and is acquired by each image sensor; the positional deviation between each image sensor to be matched and the reference image sensor is obtained based on the multiple resolution test target images acquired by each image sensor; the six-degree-of-freedom adjustment stage finely adjusts the position of each image sensor to be matched based on the deviation.

8. The prism-based multispectral camera sensor image plane matching device according to claim 7, characterized in that, The six-degree-of-freedom adjustment stage includes: The XYZ axis translation stage is used to adjust the deviations of the X, Y, and Z axes between the image sensor to be matched and the reference image sensor. The XYZ axis angular positioning stage is used to adjust the angular offsets θ(X1), θ(Y1), and θ(Z1) in the X, Y, and Z directions between the image sensor to be matched and the reference image sensor.

9. The prism-based multispectral camera sensor image plane matching device according to claim 7, characterized in that, The XYZ scanning detection stage includes: XYZ axis displacement platform, which is connected to the adapter plate; The adapter plate is equipped with the projection device, camera, telecentric lens, and resolution test target. The XYZ axis displacement platform is used to adjust the positions of the projection device, camera, telecentric lens, and resolution test target on the X, Y, and Z axes.

10. The prism-based multispectral camera sensor image plane matching device according to claim 7, characterized in that, Also includes: A prism bonding structure includes a prism mounting bracket and a sensor bonding plate, wherein the prism mounting bracket is used to fix a multi-beam splitter prism, and an image sensor is fixed on one side of the sensor bonding plate and connected to the prism mounting bracket on the other side. An adhesive plate with a reference image sensor fixed on one side is directly fixed to the corresponding position of the prism mounting bracket, so that the multi-beam prism splits the received light source and illuminates the reference image sensor. An adhesive plate with the image sensor to be matched fixed on one side is pre-fixed at the corresponding position of the prism mounting bracket, so that the multi-beam prism can split the received light source and illuminate the reference image sensor. The adhesive plate, on one side of which the image sensor to be matched is fixed, is connected to the six-degree-of-freedom adjustment stage on the other side. The position of the image sensor to be matched is adjusted under the action of the six-degree-of-freedom adjustment stage. After adjusting the position of the image sensor to be matched, fix the adhesive plate with the image sensor to be matched fixed on one side onto the prism mounting bracket.

Images