Image alignment method and device for VNIR multi-module hyperspectral camera and medium

Abstract

The invention discloses an image alignment method and device for a VNIR multi-module hyperspectral camera and a medium, and can be applied to the technical field of image data processing. According to the method, the sharpness of the acquired to-be-processed VIS image and the acquired to-be-processed NIR image is calculated, the identification image on the to-be-processed VIS image and the identification image on the to-be-processed NIR image are identified according to the sharpness, after the identification image on the to-be-processed VIS image and the identification image on the to-be-processed NIR image are subjected to identification correspondence, the identification feature points on the identification image after identification correspondence are identified, and the identification feature points on the to-be-processed VIS image and the identification feature points on the to-be-processed NIR image are identified. Performing feature point correspondence on the identification feature points and identification labels and identification positions on the to-be-processed VIS image and the to-be-processed NIR image, performing calculation according to the identification feature points corresponding to the feature points to obtain a homography matrix, and performing alignment operation on the to-be-processed VIS image and the to-be-processed NIR image according to the homography matrix to obtain the to-be-processed VIS image and the to-be-processed NIR image. Therefore, the precision of the image alignment result can be effectively improved.

Description

Technical Field

[0001] The present application relates to the technical field of image data processing, and in particular to an image alignment method, device and medium for a VNIR multi-module hyperspectral camera. Background Art

[0002] In the related technology, the VNIR multi-module hyperspectral camera is an advanced imaging device that combines hyperspectral imaging technology from visible light to near-infrared (Visible and Near-Infrared, VNIR) bands. The camera provides rich information for the application field by capturing a wide range of spectral data of different wavelengths. Among them, multiple spectral sensors are integrated in the VNIR multi-module hyperspectral camera, so it is necessary to align the spectral images obtained by multiple modules. Since the VNIR multi-module hyperspectral camera does not integrate attitude detection equipment such as gyroscopes, image alignment is currently achieved through image feature point detection methods. However, this method is limited by the accuracy of the image feature point detection results, resulting in low accuracy of the alignment results obtained in actual applications.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose an image alignment method, device and medium for a VNIR multi-module hyperspectral camera, which can effectively improve the accuracy of the image alignment results.

[0005] To achieve the above object, an embodiment of the present application provides an image alignment method for a VNIR multi-module hyperspectral camera, the method comprising the following steps:

[0006] Acquire a plurality of to-be-processed VIS images and a plurality of to-be-processed NIR images corresponding to the target photographed object photographed by the VNIR multi-module hyperspectral camera, wherein the to-be-processed VIS images and the to-be-processed NIR images both contain the target object image and the identification image;

[0007] Calculating the sharpness of the VIS image to be processed and the NIR image to be processed;

[0008] Identify the identification image on the VIS image to be processed and the NIR image to be processed according to the sharpness;

[0009] Marking and correspondingly matching the identification image on the VIS image to be processed with the identification image on the NIR image to be processed;

[0010] Identify the identification feature points on the identification image after the identification corresponds to the identification;

[0011] Matching the identification feature points with the identification numbers and identification positions on the VIS image to be processed and the NIR image to be processed respectively;

[0012] A homography matrix is ​​calculated based on the identified feature points after the feature points are aligned;

[0013] An alignment operation is performed on the VIS image to be processed and the NIR image to be processed according to the homography matrix.

[0014] In some embodiments, the identification image includes a QR code-like icon image or a preset corresponding point image, the QR code-like icon corresponding to the QR code-like icon image is set on a calibration plate, the calibration plate is set on a reference plate, and the target object is set in a hollow area of ​​the calibration plate and in contact with the reference plate.

[0015] In some embodiments, the calculating the sharpness of the to-be-processed VIS image and the to-be-processed NIR image comprises:

[0016] Processing the VIS image to be processed and the NIR image to be processed respectively to obtain corresponding Laplace images;

[0017] The Laplace variance of the Laplace image corresponding to the VIS image to be processed and the Laplace image corresponding to the NIR image to be processed is calculated, and the Laplace variance is used to measure the sharpness of the VIS image to be processed and the NIR image to be processed.

[0018] In some embodiments, the identifying the identification image on the to-be-processed VIS image and the to-be-processed NIR image according to the sharpness includes:

[0019] sorting the Laplace variances of the VIS image to be processed and the NIR image to be processed respectively;

[0020] Based on the sorting results, identification images on the to-be-processed VIS image and the to-be-processed NIR image are respectively identified.

[0021] In some embodiments, the identifying the identification images on the VIS image to be processed and the NIR image to be processed respectively based on the sorting results includes:

[0022] Performing mark detection on the to-be-processed VIS image and the to-be-processed NIR image based on the sorting result, wherein the mark detection includes threshold detection and contour detection;

[0023] The identification images of the VIS image to be processed and the NIR image to be processed are identified according to the detection results.

[0024] In some embodiments, the step of calculating a homography matrix based on the identified feature points after the feature points are aligned includes:

[0025] Iteratively select the feature point to be calculated from the identified feature points corresponding to the feature points;

[0026] The homography matrix is ​​obtained by calculation according to the feature points to be calculated.

[0027] In some embodiments, the aligning operation of the to-be-processed VIS image and the to-be-processed NIR image according to the homography matrix includes:

[0028] Selecting one type of images from the to-be-processed VIS images and the to-be-processed NIR images as the images to be transferred, and the remaining type of images as the target images;

[0029] Obtaining pixel coordinates of the image to be transferred;

[0030] The grayscale value of the image to be transferred is transferred to the target image according to the pixel coordinates and the homography matrix.

[0031] To achieve the above object, another aspect of the embodiment of the present application provides an image alignment device for a VNIR multi-module hyperspectral camera, the device comprising:

[0032] The first module is used to obtain a number of to-be-processed VIS images and a number of to-be-processed NIR images corresponding to the target photographed object photographed by the VNIR multi-module hyperspectral camera, wherein the to-be-processed VIS images and the to-be-processed NIR images both contain the target object image and the identification image;

[0033] The second module is used to calculate the sharpness of the VIS image to be processed and the NIR image to be processed;

[0034] A third module is used to identify the identification image on the VIS image to be processed and the NIR image to be processed according to the sharpness;

[0035] A fourth module is used to mark and correspond the identification image on the VIS image to be processed with the identification image on the NIR image to be processed;

[0036] A fifth module is used to identify the identification feature points on the identification image after the identification corresponds;

[0037] A sixth module is used to correspond the identification feature points to the identification numbers and identification positions on the VIS image to be processed and the NIR image to be processed respectively;

[0038] The seventh module is used to calculate the homography matrix according to the identified feature points after the feature points are aligned;

[0039] An eighth module is used to align the VIS image to be processed and the NIR image to be processed according to the homography matrix.

[0040] To achieve the above objective, another aspect of the present application provides a computer device, including:

[0041] at least one processor;

[0042] at least one memory for storing at least one program;

[0043] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0044] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0045] The embodiments of the present application include at least the following beneficial effects: The present application provides an image alignment method, device and medium for a VNIR multi-module hyperspectral camera. The scheme obtains several to-be-processed VIS images and several to-be-processed NIR images containing target object images and identification images, calculates the sharpness of the to-be-processed VIS images and the to-be-processed NIR images, and identifies the identification images on the to-be-processed VIS images and the to-be-processed NIR images according to the sharpness, identifies the identification images on the to-be-processed VIS images and the to-be-processed NIR images after identification correspondence, identifies the identification feature points on the identification images after identification correspondence, and respectively matches the identification feature points with the identification numbers and identification positions on the to-be-processed VIS images and the to-be-processed NIR images, calculates a homography matrix based on the identification feature points corresponding to the feature points, and then performs alignment operations on the to-be-processed VIS images and the to-be-processed NIR images based on the homography matrix, thereby realizing the alignment process of the VIS images and the NIR images without relying on the recognition results of the target image feature points, and effectively improving the accuracy of the image alignment results. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a flow chart of an image alignment method for a VNIR multi-module hyperspectral camera provided in an embodiment of the present application;

[0047] Figure 2 is a schematic diagram of a calibration plate provided in an embodiment of the present application;

[0048] Figure 3 is a schematic diagram of shooting using a calibration plate provided in an embodiment of the present application;

[0049] Figure 4 It is a VIS image corresponding to a wavelength of 510 nm provided in an embodiment of the present application;

[0050] Figure 5 It is a VIS image corresponding to a wavelength of 590nm provided in an embodiment of the present application;

[0051] Figure 6 It is a VIS image corresponding to a wavelength of 690nm provided in an embodiment of the present application;

[0052] Figure 7 It is the NIR image corresponding to the wavelength of 713nm provided in the embodiment of the present application;

[0053] Figure 8 It is the NIR image corresponding to the wavelength of 828nm provided in the embodiment of the present application;

[0054] Fig. 9 It is the NIR image corresponding to the wavelength of 920nm provided in the embodiment of the present application;

[0055] Fig.10 is a schematic diagram of ArUco icon recognition results provided in an embodiment of the present application;

[0056] Fig.11 The VIS image provided by the embodiment of the present application is Fig.12 Schematic diagram of the corresponding icon feature points of the NIR image shown;

[0057] Fig.12 The NIR image provided by the embodiment of the present application is Fig.11 Schematic diagram of the icon feature points corresponding to the VIS image shown;

[0058] Fig.13 The VIS image to be processed provided by the embodiment of the present application is Fig.14 The schematic diagram of the result of the alignment of the NIR images to be processed is shown;

[0059] Fig.14 The embodiment of this application provides Fig.13 The schematic diagram of the result of aligning the VIS image to be processed to the NIR image to be processed is shown;

[0060] Fig.15 It is a module schematic diagram of an image alignment device for a VNIR multi-module hyperspectral camera provided in an embodiment of the present application;

[0061] Fig.16 It is a schematic diagram of the hardware structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application, they are only examples of devices and methods consistent with some aspects of the embodiments of the present application.

[0063] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0064] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0066] In the related technology, the VNIR multi-module hyperspectral camera is an advanced imaging device that combines hyperspectral imaging technology from visible light to near-infrared (Visible and Near-Infrared, VNIR) bands, and provides rich information for application fields by capturing spectral data of a wide range of different wavelengths. Specifically, the VNIR multi-module hyperspectral camera can cover the spectral range from visible light to near-infrared (wavelength range of about 510nm-920nm), capturing data of multiple wavelengths on each pixel, which enables users to perform fine spectral analysis.

[0067] It is understandable that the VNIR multi-module design allows the integration of multiple sensors and spectral modules. The current mainstream spectral cameras generally integrate two spectral sensing modules, covering the VIS (visible light) band and NIR (near infrared) band spectra respectively. The system then integrates the information of the two spectral sensing modules to uniformly output the hyperspectral information of the photographed object. Since the VNIR multi-module hyperspectral camera does not integrate attitude detection equipment such as gyroscopes, image alignment is currently achieved through image feature point detection methods. However, this method is limited by the accuracy of the image feature point detection results, resulting in low accuracy of the alignment results obtained in actual applications.

[0068] In view of this, an embodiment of the present application provides an image alignment method, device and medium for a VNIR multi-module hyperspectral camera. The present application obtains a number of to-be-processed VIS images and a number of to-be-processed NIR images containing target object images and identification images, calculates the sharpness of the to-be-processed VIS images and the to-be-processed NIR images, and identifies the identification images on the to-be-processed VIS images and the to-be-processed NIR images according to the sharpness. After the identification images on the to-be-processed VIS images are identified and corresponded with the identification images on the to-be-processed NIR images, the identification feature points on the identification images after the identification correspondence are identified, and the identification feature points are respectively corresponded with the identification numbers and identification positions on the to-be-processed VIS images and the to-be-processed NIR images. A homography matrix is ​​calculated based on the identification feature points corresponding to the feature points, and then the to-be-processed VIS images and the to-be-processed NIR images are aligned according to the homography matrix, so that the alignment process of the VIS images and the NIR images can be realized without relying on the recognition results of the target image feature points, and the accuracy of the image alignment results is effectively improved.

[0069] The image alignment method for VNIR multi-module hyperspectral camera provided in the embodiment of the present application relates to the field of image data processing technology. The image alignment method for VNIR multi-module hyperspectral camera provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. The cloud server, the server can also be a node server in the blockchain network; the software can be an application that implements the image alignment method for VNIR multi-module hyperspectral camera, etc., but is not limited to the above form.

[0070] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0071] Figure 1 is an optional flow chart of an image alignment method for a VNIR multi-module hyperspectral camera provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S110 to S180:

[0072] Step S110, obtaining a plurality of to-be-processed VIS images and a plurality of to-be-processed NIR images corresponding to the target photographed object photographed by the VNIR multi-module hyperspectral camera, wherein both the to-be-processed VIS images and the to-be-processed NIR images contain the target object image and the identification image;

[0073] Step S120, calculating the sharpness of the to-be-processed VIS image and the to-be-processed NIR image;

[0074] Step S130, identifying the identification image on the to-be-processed VIS image and the to-be-processed NIR image according to the sharpness;

[0075] Step S140, marking and correspondingly matching the identification image on the VIS image to be processed with the identification image on the NIR image to be processed;

[0076] Step S150, identifying the marking feature points on the marking image corresponding to the marking;

[0077] Step S160, matching the identification feature points with the identification numbers and identification positions on the to-be-processed VIS image and the to-be-processed NIR image respectively;

[0078] Step S170, calculating a homography matrix according to the marked feature points after the feature points are aligned;

[0079] Step S180: aligning the VIS image to be processed and the NIR image to be processed according to the homography matrix.

[0080] It is understandable that the VNIR multi-module hyperspectral camera used in this embodiment has three sets of lenses, which are used to capture VIS band (510nm-690nm), NIR band (713nm-920nm) and RGB information respectively. When the camera is operated, it can take pictures of the same object at the same time and save the images of the corresponding bands. Due to the position differences of the three sets of lenses, when the VNIR multi-module hyperspectral camera takes pictures of the same object, the imaging position will be different, so it is necessary to align the images of different bands, so as to provide effective data for subsequent spectral analysis.

[0081] In an embodiment of the present application, the identification image may be a QR code icon (ArUco) image. The QR code icon corresponding to the QR code icon image is set on a calibration plate. The calibration plate is set on a reference plate. When the spectral camera is used to capture images, the target object is set in the hollow area of ​​the calibration plate and in contact with the reference plate. The QR code icon is used to record the information of the calibration plate. Specifically, the ArUco icon is a two-dimensional matrix marking pattern specially used in the field of computer vision. It is similar to a QR code, but different from a QR code in function. The ArUco icon is designed to help computers identify and locate objects. Figure 2 As shown, the ArUco icon is designed as a simple geometric shape composed of black and white squares. Compared with the QR code, it is easy to identify and detect, and can be accurately identified even in darker or more complex environments. Each ArUco icon contains a unique identification number. This embodiment can detect the icon and identify its identification number and key point position through a computer vision algorithm.

[0082] It is understandable that if Figure 2 As shown, this embodiment can generate ArUco icons with different identification numbers and place them on the calibration board according to a preset combination method ( Figure 2ArUco icons at the four top corners of the calibration plate). The combination of identification numbers corresponding to the ArUco icons can be used as the marking number of the photographed object. Specifically, this embodiment can use DICT_7X7_1000 (7×7 means that each icon is a 7×7 matrix; 1000 means that this dictionary contains 1000 different icons, and the identification numbers are from 0 to 999) to construct the ArUco icon, taking into account the amount of information and the simplicity of the icon. When constructing the calibration plate, a certain number of ArUco icons are placed in the corners of the calibration plate, and the middle of the calibration plate is hollowed out to place the photographed object and the shooting reference plate. Generally, the present invention places 4 ArUco icons of 2×2 in the four corners of the calibration plate, and leaves a certain width of white around each icon to increase the stability of recognition. The identification number of the icon can be a randomly generated different identification number, or it can be generated according to a combination with a certain meaning.

[0083] After the design of the calibration plate is completed, when using the VNIR multi-module hyperspectral camera to shoot, such as Figure 3 As shown, place the reference plate at the bottom, place the calibration plate on it, and then place the photographed object in the hollowed-out part of the calibration plate. Ensure that the calibration plate, reference plate, and photographed object can be photographed at the same time during shooting. Specifically, in this embodiment, the VIS lens and NIR lens in the VNIR multi-module hyperspectral camera can be used to simultaneously shoot VIS images (VIS images to be processed) and NIR images (NIR images to be processed). Among them, the VIS lens and NIR lens of this embodiment can respectively capture one or more images in the range of 510nm-690nm and 713nm-920nm. For example, Figure 4 (510nm), Figure 5 (590nm) and Figure 6 (690nm) as shown in the VIS image, and Figure 7 (713nm), Figure 8 (828nm) and Fig. 9 (920nm) shown in NIR image. Figures 4 to 9 It can be seen that this embodiment ensures that the calibration plate, the photographed object and the reference plate can be completely captured in both the VIS image and the NIR image during shooting.

[0084] In the embodiment of the present application, for the pictures taken by the VNIR multi-module hyperspectral camera, suitable pictures need to be selected from the VIS group and the NIR group as the VIS image to be processed and the NIR image to be processed, and then the VIS image to be processed and the NIR image to be processed are aligned. In the image selection process, it is necessary to ensure that the selected images can correctly identify the ArUco icon in order to improve the accuracy of the image alignment algorithm.

[0085] Specifically, after obtaining the VIS image to be processed and the NIR image to be processed, this embodiment measures the edge strength of the image by calculating the Laplacian transform of the image, thereby judging the sharpness of the image. Among them, the sharpness of the image is closely related to the clarity of its edge. The clearer the edge, the sharper the image. It is generally believed that the sharper the image, the greater the difference between the ArUco icon and the background, and the more conducive to the correct identification of the ArUco icon. It can be understood that this embodiment can calculate the Laplace variance of the Laplace image corresponding to the VIS image to be processed and the Laplace image corresponding to the NIR image to be processed after processing the VIS image to be processed and the NIR image to be processed respectively. The Laplace variance of this embodiment is used to measure the sharpness of the VIS image to be processed and the NIR image to be processed.

[0086] Specifically, the process of Laplace image transformation is as follows:

[0087]

[0088] In the formula, I(x,y) represents the grayscale value of the VIS image or NIR image to be processed, x and y represent the spatial coordinates of the image, and L(x,y) represents the Laplacian image. In the Laplacian image, the larger the absolute value of the pixel value, the greater the edge strength of the corresponding position.

[0089] The calculation process of Laplace variance is as follows:

[0090]

[0091] M and N represent the width and height of the Laplace image; L(i,j) represents the pixel value of the Laplace image, μ represents the mean value of the Laplace image, and Variance represents the Laplace variance. Specifically, the larger the variance, the greater the change in edge intensity in the image, and the sharper the image; the smaller the variance, the blurrier the image.

[0092] In an embodiment of the present application, after obtaining the Laplace variance representing the sharpness, the Laplace variances of the VIS image to be processed and the NIR image to be processed are sorted respectively, and the identification images on the VIS image to be processed and the NIR image to be processed are identified based on the sorting results. Taking the identification of ArUco icons as an example, this embodiment can sort the images of the VIS group and the NIR group from large to small according to the variance calculated in the previous step, and use the ArUco icon recognition algorithm to identify the ArUco icons in the images until the ArUco icons can be correctly identified in both the VIS group and the NIR group. If in some cases, due to the image shooting quality or other factors, the ArUco icon cannot be correctly identified, this embodiment can also achieve image alignment by setting key points in the interface. Among them, the identification image corresponding to the key point is a preset corresponding point image.

[0093] Specifically, when identifying the identification image corresponding to the ArUco icon, the present embodiment can perform mark detection on the VIS image to be processed and the NIR image to be processed based on the sorting results, and identify the identification images of the VIS image to be processed and the NIR image to be processed according to the detection results. Among them, mark detection includes threshold detection and contour detection. It can be understood that threshold detection can be to apply binarization threshold processing to the grayscale image of the spectrum to highlight the marked area. Contour detection can be to identify contours in the binary image, which are potential marker candidates, and then simplify and filter these contours according to the shape, find quadrilaterals, and calculate the corner points of the quadrilaterals. When the threshold detection and contour detection are completed, the quadrilateral is perspectively transformed using the corner points of the quadrilateral to distort it into a square so that the algorithm can standardize the size and direction of the marker. Then extract the binary code inside the marker. The interior of the ArUco marker is composed of a grid of black and white cells, and each cell code is represented by a binary value. The extracted binary code is matched with the predefined ArUco marker dictionary to identify the number of the marker, thereby obtaining Fig.10 Recognition results of the ArUco icon shown.

[0094] In the embodiment of the present application, after the ArUco icon is correctly identified, the identification image corresponding to the ArUco icon is matched according to the identification number in the ArUco icon. Then, the points on the upper left, upper right, lower left, and lower right corners of the ArUco icon contained in the VIS image to be processed and the NIR image to be processed are detected as identification feature points. In this example, there are 16 ArUco icons, with a total of 64 feature points to be processed, and the positions of these identification feature points are recorded. Then, the identification feature points detected in the VIS image and the NIR image are matched one by one according to the ArUco icon identification number and position (such as Fig.11 and Fig.12 corresponding process).

[0095] In an embodiment of the present application, after obtaining the correspondence between the feature points of the VIS image and the NIR image, the RANSAC (random sampling consensus) algorithm is used to estimate the homography matrix. Specifically, the homography matrix can be calculated based on the feature points to be calculated after iteratively selecting the feature points to be calculated from the identification feature points corresponding to the feature points. Exemplarily, this embodiment can iteratively select a portion of corresponding points through the RANSAC algorithm and use these points to estimate the homography matrix, thereby finding the best transformation model. Among them, as shown in the following formula, the homography matrix H is a 3×3 matrix that represents the transformation relationship between the two images:

[0096]

[0097] After estimating the homography transformation matrix, this embodiment can use the homography matrix to transform one image to another image, thereby achieving image alignment. Specifically, this embodiment obtains a homography matrix that aligns the VIS image to the NIR image.

[0098] It can be understood that the present embodiment selects one type of image from the VIS image to be processed and the NIR image to be processed as the image to be transferred, and the remaining type of image is used as the target image, and then obtains the pixel coordinates of the image to be transferred, and transfers the grayscale value of the image to be transferred to the target image according to the pixel coordinates and the homography matrix. Specifically, the present real-time exchange rate can use the image perspective transformation algorithm to apply the homography matrix to the image to be aligned, so as to convert each pixel in the image to be aligned to the corresponding position in the target image. Exemplarily, taking the alignment of the VIS image to be processed to the NIR image as an example, the present embodiment first multiplies the coordinates of each pixel of the VIS image to be processed by the homography matrix to obtain the corresponding coordinates x', y', and transfers the grayscale value of the corresponding pixel of the VIS image to be processed to obtain Fig.13 The VIS image to be processed is shown Fig.14 In this embodiment, for a plurality of VIS band images, the obtained homography matrix can be used to act on each VIS band image to achieve the alignment of each VIS band image to the NIR band image.

[0099] The embodiment of the present application accurately aligns the VIS and NIR image information so that the information of corresponding positions of the two sets of images can correspond to each other. Image data of different spectra can be directly fused, analyzed and processed, thereby providing richer visual data and providing effective data support for subsequent data analysis and feature extraction.

[0100] Specifically, the practical applications of the VIS and NIR dual-lens camera device include but are not limited to:

[0101] Industrial inspection: In the manufacturing industry, the VIS and NIR dual-lens system can simultaneously detect product surface and internal defects. The aligned image data can be superimposed for comprehensive analysis to improve the accuracy and efficiency of inspection.

[0102] Medical imaging: In the field of medical imaging, by aligning VIS and NIR images, we can better observe the condition of the tissue under the skin and assist in the diagnosis of various diseases such as skin lesions. The combination of information from the two spectra can provide a more comprehensive basis for diagnosis.

[0103] Food detection: Using the VIS and NIR dual-lens system, spectral imaging information can be used more effectively to detect ingredient content, contaminants and deterioration in food to ensure food safety and quality. After adding VIS and NIR image alignment technology, small particles such as tea leaves and coffee beans can be detected more effectively.

[0104] It can be seen that, through the image alignment technology, the VIS and NIR dual-lens camera device in this embodiment can more effectively combine the two spectral information to achieve more efficient and accurate multi-spectral imaging applications. This plays an important role in improving the quality of data analysis and enhancing the application effects in multiple fields.

[0105] Reference Fig.15 The embodiment of the present application provides an image alignment device for a VNIR multi-module hyperspectral camera, the device comprising:

[0106] The first module 1610 is used to obtain a plurality of to-be-processed VIS images and a plurality of to-be-processed NIR images corresponding to the target photographed object photographed by the VNIR multi-module hyperspectral camera, wherein both the to-be-processed VIS images and the to-be-processed NIR images contain the target object image and the identification image;

[0107] The second module 1620 is used to calculate the sharpness of the VIS image to be processed and the NIR image to be processed;

[0108] The third module 1630 is used to identify the identification image on the VIS image to be processed and the NIR image to be processed according to the sharpness;

[0109] The fourth module 1640 is used to mark and correspond the identification image on the VIS image to be processed with the identification image on the NIR image to be processed;

[0110] The fifth module 1650 is used to identify the marking feature points on the marking image after the marking corresponds;

[0111] The sixth module 1660 is used to correspond the identification feature points to the identification numbers and identification positions on the VIS image to be processed and the NIR image to be processed respectively;

[0112] The seventh module 1670 is used to calculate a homography matrix according to the marked feature points after the feature points are aligned;

[0113] The eighth module 1680 is used to align the VIS image to be processed and the NIR image to be processed according to the homography matrix.

[0114] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0115] The embodiment of the present application also provides a computer device, the computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned image alignment method for a VNIR multi-module hyperspectral camera when executing the computer program. The computer device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0116] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0117] See also Fig.16 , Fig.16 The hardware structure of a computer device according to another embodiment is shown, and the computer device includes:

[0118] The processor 1710 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0119] The memory 1720 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1720 can store an operating system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1720, and the processor 1710 calls and executes the image alignment method for a VNIR multi-module hyperspectral camera in the embodiment of the present application;

[0120] Input / output interface 1730, used to implement information input and output;

[0121] Communication interface 1740, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0122] Bus 1750 , which transmits information between various components of the device (e.g., processor 1710 , memory 1720 , input / output interface 1730 , and communication interface 1740 );

[0123] The processor 1710 , the memory 1720 , the input / output interface 1730 , and the communication interface 1740 are connected to each other in communication within the device via a bus 1750 .

[0124] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned image alignment method for a VNIR multi-module hyperspectral camera.

[0125] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0126] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0127] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0128] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0129] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0130] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0131] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0132] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0133] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0134] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0135] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0136] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0137] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. An image alignment method for a VNIR multi-module hyperspectral camera, characterized in that: The method comprises the following steps: Acquire a plurality of to-be-processed VIS images and a plurality of to-be-processed NIR images corresponding to the target photographed object photographed by the VNIR multi-module hyperspectral camera, wherein the to-be-processed VIS images and the to-be-processed NIR images both contain the target object image and the identification image; Calculating the sharpness of the VIS image to be processed and the NIR image to be processed; Identify the identification image on the VIS image to be processed and the NIR image to be processed according to the sharpness; Marking and correspondingly matching the identification image on the VIS image to be processed with the identification image on the NIR image to be processed; Identify the identification feature points on the identification image after the identification corresponds to the identification; Matching the identification feature points with the identification numbers and identification positions on the VIS image to be processed and the NIR image to be processed respectively; A homography matrix is ​​calculated based on the identified feature points after the feature points are aligned; An alignment operation is performed on the VIS image to be processed and the NIR image to be processed according to the homography matrix.

2. The method according to claim 1, characterized in that The identification image includes a QR code-like icon image or a preset corresponding point image, the QR code-like icon corresponding to the QR code-like icon image is set on a calibration plate, the calibration plate is set on a reference plate, and the target photographic object is set in a hollow area of ​​the calibration plate and in contact with the reference plate.

3. The method according to claim 1, characterized in that The calculating the sharpness of the to-be-processed VIS image and the to-be-processed NIR image comprises: Processing the VIS image to be processed and the NIR image to be processed respectively to obtain corresponding Laplace images; The Laplace variance of the Laplace image corresponding to the VIS image to be processed and the Laplace image corresponding to the NIR image to be processed is calculated, and the Laplace variance is used to measure the sharpness of the VIS image to be processed and the NIR image to be processed.

4. The method according to claim 3, characterized in that The identifying the identification image on the to-be-processed VIS image and the to-be-processed NIR image according to the sharpness includes: sorting the Laplace variances of the VIS image to be processed and the NIR image to be processed respectively; Based on the sorting results, identification images on the to-be-processed VIS image and the to-be-processed NIR image are respectively identified.

5. The method according to claim 4, characterized in that The step of respectively identifying the identification images on the VIS image to be processed and the NIR image to be processed based on the sorting results includes: Performing mark detection on the to-be-processed VIS image and the to-be-processed NIR image based on the sorting result, wherein the mark detection includes threshold detection and contour detection; The identification images of the VIS image to be processed and the NIR image to be processed are identified according to the detection results.

6. The method according to claim 1, characterized in that The step of calculating a homography matrix based on the identified feature points after the feature points correspond to each other includes: Iteratively select the feature point to be calculated from the identified feature points corresponding to the feature points; The homography matrix is ​​obtained by calculation according to the feature points to be calculated.

7. The method according to claim 1, characterized in that The aligning operation of the VIS image to be processed and the NIR image to be processed according to the homography matrix includes: Selecting one type of images from the to-be-processed VIS images and the to-be-processed NIR images as the images to be transferred, and the remaining type of images as the target images; Obtaining pixel coordinates of the image to be transferred; The grayscale value of the image to be transferred is transferred to the target image according to the pixel coordinates and the homography matrix.

8. An image alignment device for a VNIR multi-module hyperspectral camera, characterized in that: The device comprises: The first module is used to obtain a number of to-be-processed VIS images and a number of to-be-processed NIR images corresponding to the target photographed object photographed by the VNIR multi-module hyperspectral camera, wherein the to-be-processed VIS images and the to-be-processed NIR images both contain the target object image and the identification image; The second module is used to calculate the sharpness of the VIS image to be processed and the NIR image to be processed; A third module is used to identify the identification image on the VIS image to be processed and the NIR image to be processed according to the sharpness; A fourth module is used to mark and correspond the identification image on the VIS image to be processed with the identification image on the NIR image to be processed; A fifth module is used to identify the identification feature points on the identification image after the identification corresponds; A sixth module is used to correspond the identification feature points to the identification numbers and identification positions on the VIS image to be processed and the NIR image to be processed respectively; The seventh module is used to calculate the homography matrix according to the identified feature points after the feature points are aligned; An eighth module is used to align the VIS image to be processed and the NIR image to be processed according to the homography matrix.

9. A computer device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images