Cooperative target-based public region extraction method and system for infrared camera array

Abstract

The application provides a common area extraction method of an infrared camera array based on cooperative targets, cooperative target selection is performed, cooperative targets adopt infrared sources, and the wavelength band of the infrared light generated is in or covers the overlapping part of the corresponding wavelength band of the infrared camera array; cooperative target placement is performed, three cooperative targets are installed on the same plane, the distances among the three cooperative targets are not equal in the image captured by the camera array, and the distance ratios among the three cooperative targets are determined; the infrared camera array simultaneously captures and samples the three cooperative targets, converts the three cooperative targets into black and white images, removes interference noise points, obtains the white areas corresponding to the three cooperative targets respectively, and then obtains three points by calculating the mean values of the connected domain coordinates; the corresponding relationship of the three points in the image is obtained through the different distances among the three cooperative targets, affine transformation is performed, and the image after the affine transformation is traversed according to pixels, and the image overlapping part is cut out as the final common area result.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of digital image registration application processing, in particular to a common area acquisition technical scheme of an infrared camera array based on a cooperative target. BACKGROUND

[0002] Image registration is a popular field of current computer graphics research, attracting more and more researchers' attention, and being gradually applied in medical, military, face recognition and other fields. Image registration can be divided into non-cooperative target registration and cooperative target-based registration. Among them, the cooperative target is to use the characteristics that the objects with identity are easy to detect and identify in the image to provide reference for image registration.

[0003] The traditional non-cooperative target registration method includes gray registration, mutual information registration, feature point registration, multi-modal registration and deep learning method, but this kind of method needs more calculation time and cannot be selected in some high real-time system environment.

[0004] In the current process of improving the automation of environmental monitoring in China, the image registration problem in the application scene of motor vehicle exhaust detection needs to be solved. The Chinese Academy of Sciences and Wuhan University cooperated to carry out the national key research and development plan: technical method system research on mobile pollution source emission on-site law enforcement supervision.

[0005] The research and development plan needs to detect various pollution gases, so a plurality of infrared cameras are used to form an array, and different cameras detect different pollutants. However, due to the different positions of the cameras, there is a certain positional deviation in the images obtained by the camera array.

[0006] In addition, the infrared camera array has different positional relationships with the shooting object in different application occasions. In actual application scenarios, the position of the infrared camera array changes with the installation conditions. In actual application, the infrared camera is installed at a height of 4m-6m, the lens is downward, and the vehicle driving below is shot. The infrared camera array is roughly parallel to the ground. The small deviation in the installation process and the problem of uneven road surface may cause the infrared camera array to be non-parallel to the shooting target. In addition, the infrared camera is equipped with different waveband filters, which causes great differences in brightness distribution and even sharpness of the pictures shot by different cameras. The concentration inversion of the research and development plan needs to obtain a unified common area through registration. The registered infrared image is used for quantitative calculation of air pollutant content. The accuracy of registration will affect the calculation accuracy of the research and development plan for air pollutant concentration inversion. Compared with the deep learning registration and multi-modal registration method, the registration accuracy of the present application is high and can meet the needs of the research and development plan for accuracy.

[0007] REFERENCE Figure 1(Wherein the black box 1-5 corresponding to the five infrared camera acquisition image), motor vehicle exhaust detection research and development plan using five infrared camera with the same specifications and parameters of an infrared camera array, five infrared cameras placed in a metal frame, the center axis of the infrared camera lens parallel to each other, the lens are located in the same plane. Infrared camera array shot object (motor vehicle exhaust) and infrared camera four to five meters apart, but due to the position of the infrared camera has some different, which leads to the same shot object (motor vehicle) in five infrared camera shot image position is different, in addition, different infrared camera corresponding to the filter wavelength is also different, which will lead to the picture in brightness distribution and even sharpness have a big gap, and deep learning and traditional registration method requires a large amount of calculation and time-consuming, because need for research and development plan to provide a same area for subsequent concentration inversion, and need to meet the real-time processing needs, it is necessary to solve the real-time image registration problem, get the common area of infrared camera array image.

[0008] Therefore, the related research gap needs to be filled in the field, which is suitable for camera array position with mobility, camera parameters are not the same scene, for example, the specific scene design is: multi-infrared camera array acquisition needs, the position of the infrared camera in the array should meet the camera lens are located in the same plane, and the center axis of the infrared camera should be parallel to each other, the included angle should be less than 5°, the size of the camera view angle, the field of view angle, the camera shooting optical band is different (the same scene is also applicable). The scene needs to have the condition of installing infrared cooperative target (infrared source), and the scene has high real-time requirement, and the traditional deep learning, multi-modal registration method cannot meet the requirement.

[0009] By using infrared source as cooperative target to complete registration to obtain the position relationship of camera array shot image, the requirement of high real-time system is met. The registered infrared image is used for quantitative calculation of air pollutant content. The accuracy of registration will affect the calculation accuracy of air pollutant concentration inversion of the research and development plan, and the solution needs to be accurate, real-time, small in calculation amount, and meet the requirements of accuracy and real-time of application scene. SUMMARY

[0010] According to the requirements of real-time, accuracy, mobility and convenience of application scene, the application proposes a technical solution for registering the images collected by the infrared camera array to obtain the maximum common area in the imaging of different cameras.

[0011] In order to achieve the above purpose, the application proposes a common area extraction method of infrared camera array based on cooperative target, which performs the following processing,

[0012] The selection of cooperative target includes setting the cooperative target to use infrared source, and the wavelength band of the generated infrared light is in or covers the overlapping part of the corresponding wavelength band of the infrared camera array;

[0013] The placement of the cooperative targets is performed, including installing three cooperative targets on the same plane, the distances among the three cooperative targets are not equal in the images captured by the camera array, and the distance ratios among the three cooperative targets are determined;

[0014] The three cooperative targets are simultaneously captured by the infrared camera array, and the same number of images as the number of cameras in the infrared camera array is obtained;

[0015] The collected images are converted into black-and-white images through gray-scale binaryzation, the interference noise points formed in the imaging are removed through a morphological erosion operation, so as to obtain the white regions corresponding to the three cooperative targets respectively, and then the connected domain coordinate mean values are obtained through connected domain labeling to obtain three points;

[0016] The corresponding relationship of the three points in the multiple images captured by the infrared camera array is obtained through the different distances among the three cooperative targets, and the affine transformation is performed through the corresponding relationship of the three points among different images;

[0017] The images processed through the affine transformation are traversed pixel by pixel, and the overlapping part of the images is cut out as the final common region result.

[0018] Furthermore, when the cooperative targets are placed, the three infrared sources are installed on the same plane and arranged in a right-angled triangle, and the distance ratios among the three infrared sources are 3:4:5.

[0019] Furthermore, the infrared camera array on the same plane simultaneously samples the three infrared sources on the same plane.

[0020] Furthermore, the brightness of the infrared sources in the collected images is much greater than the environment, and the gray-scale image is converted into a black-and-white image through gray-scale threshold binaryzation;

[0021] According to the imaging characteristics of the infrared camera, the black-and-white image is processed through erosion and expansion to remove the interference noise points formed in the imaging;

[0022] According to the volume and shape characteristics of the infrared source, the coordinate values of the connected domain of the black-and-white image are averaged to obtain the reference point coordinates of the three cooperative targets, and the corresponding relationship among the three points among different images is obtained through the distance relationship among the three points.

[0023] Furthermore, affine transformation is performed based on the three reference points of the images obtained by the camera array, and then the overlapping region is obtained to obtain the maximum common region.

[0024] Furthermore, it is used for detecting the concentration of motor vehicle exhaust.

[0025] In another aspect, the present application also provides a common area extraction system based on a cooperative target of an infrared camera array, comprising a processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a common area extraction method based on a cooperative target of an infrared camera array as described above.

[0026] In another aspect, the present application also provides a common area extraction system based on a cooperative target of an infrared camera array, characterized in that: comprising a readable storage medium, the readable storage medium has a computer program stored thereon, and the computer program implements a common area extraction method based on a cooperative target of an infrared camera array as described in any one of the above aspects when executed.

[0027] Image registration can be divided into cooperative target-based registration and non-cooperative target-based registration. The traditional non-cooperative target-based registration does not require additional devices, but has a large time complexity and poor real-time performance. The present application performs registration through the cooperative target approach, has high real-time performance, simple operation, light device, meets the needs of infrared camera array mobility, and has real-time processing and other characteristics. Through one-time positioning of the cooperative target, the position relationship of the images obtained by the camera array and the common area can be quickly obtained, and the subsequent repeated registration work is reduced, greatly improving the work efficiency.

[0028] The present application has the advantages of simple and convenient implementation, strong practicability, solving the problems of low practicability and inconvenience in actual application of related technologies, improving user experience, and having important market value. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 An example diagram of an infrared camera array shooting image in the prior art;

[0030] Figure 2 A schematic diagram of placement of a cooperative target in an embodiment of the present application;

[0031] Figure 3 An example diagram of sampling of a cooperative target by an infrared camera array in an embodiment of the present application;

[0032] Figure 4 A schematic diagram of acquisition of a common area after affine transformation in an embodiment of the present application;

[0033] Figure 5 A flowchart of an embodiment of the present application;

[0034] Figure 6 A cooperative target sampling diagram in an embodiment of the present application. DETAILED DESCRIPTION

[0035] The technical solution of the present application will be specifically described below in combination with the drawings and embodiments.

[0036] Referring toFigure 5 The embodiment of the present application provides a public region extraction method of an infrared camera array based on a cooperative target, which comprises the following processing steps:

[0037] Step S1, selection of the cooperative target, comprising setting the cooperative target as a small-volume infrared source, and the infrared light generated by the infrared source should be in or cover the overlapping part of the corresponding wave band of the infrared camera array (that is, the infrared light generated by the infrared source can be captured by each infrared camera in the infrared camera array, and the wave band of the infrared light does not exceed the detection range of the infrared camera);

[0038] In the present application, the cooperative target adopts an infrared source sensitive to the infrared camera, and the infrared source can generate infrared light with light intensity much greater than that of the environment, and has the characteristics of small volume and light weight, and is easy to carry.

[0039] Specifically, in step S1, the characteristics of the infrared camera are sensitive to the infrared light in a specific spectral wave band, and the area and length of the wave band are generally determined by the physical characteristics of the infrared camera itself and the parameters of the optical filter, so the selection of the cooperative target should be an infrared source sensitive to the infrared camera. The infrared source has two choices of heat source and light source. The infrared light generated by the heat source can cover a larger area of the wave band, and the cost is relatively low and easy to obtain, but considering the existence of the thermal radiation effect, the positioning of the cooperative target will be inaccurate, so a small-volume heat source needs to be selected, and the volume of the heat source should not be too small (preferably, the volume is not less than 1cm 3 ), to ensure that the cooperative target can be detected by the camera. The light source has a small light divergence angle, and the positioning of the cooperative target will be more accurate, but the cost is relatively high.

[0040] The present application requires at least three cooperative targets, which can reduce errors by referring to multiple cooperative targets, and can also meet the needs of the affine transformation in step S6.

[0041] Step S2, placement of the cooperative target, the number of cooperative targets is three. The three infrared sources are installed on the same plane and arranged in a right-angled triangle, and the ratio of the right-angled side is consistent with the height-width ratio of the camera frame, for example, for a camera with a height-width ratio of 3:4, the distance ratio between the three infrared sources is 3:4:5.

[0042] Further, in specific implementation, the infrared camera array is arranged on the same plane, and the three infrared sources are arranged on the same plane, and the two planes can have a certain angle and do not need to be completely parallel.

[0043] Reference is made to Figure 2, the embodiment sets the three cooperation targets in step S2 to be a triangle in the same plane, fixed on the same metal plate, and the distances between the three cooperation targets are different, for example, the distance between infrared source 1 and infrared source 2, the distance between infrared source 1 and infrared source 3, and the distance between infrared source 2 and infrared source 3 are 24 cm, 32 cm and 40 cm respectively (in actual implementation, the distances can be adjusted according to the shooting distance and the size of the cooperation target, but two conditions a. the distances between the three cooperation targets in the image captured by the camera array are not equal, and b. the three cooperation targets must be located in the field of view of all the cameras).

[0044] To reduce the calibration error of the cooperation target, the infrared source should be small in volume, portable and mobile, and the infrared light intensity generated by the infrared source should be much greater than the ambient infrared light, so that the gray value of the area where the cooperation target is located is much greater than the ambient value when the infrared camera array samples it, and the infrared source has the characteristics of high contrast and high signal-to-noise ratio.

[0045] In step S2, the scene captured by the infrared camera array needs to be determined first, that is, the positional relationship between the target captured by the infrared camera array and the infrared camera, including the distance and angle between them.

[0046] Referring to Figure 6 The lenses of the infrared camera array need to be in the same plane, and the camera central axes need to be parallel to each other, the focal lengths of the multiple cameras should be equal in length, and the field angles should be basically the same. In actual application, it is preferred that the infrared camera be installed at a height of 4 m-6 m, with the lens downward to capture the vehicles driving below, and the infrared camera array is roughly parallel to the ground. Small deviations in the installation process and uneven road surfaces may cause the infrared camera array to be non-parallel to the captured target, and the technical solution of the present application is applicable to both parallel and non-parallel cases. The number of cameras is two or more. The cooperation target is also located in the same plane, and then the metal plate on which the cooperation target is located is placed on the position of the captured target determined above, and all the cameras in the infrared camera array capture it to obtain a corresponding number of images.

[0047] During the shooting process, there should be no other high-power infrared light source except the cooperation target in the shooting scene, and no object that can generate infrared light with high light intensity such as high-temperature object. The cooperation target in the captured image should have high brightness, and the background should have low brightness.

[0048] The present application can be applied to the cases where the infrared camera array is parallel and non-parallel to the shooting area, but requires that the infrared camera array be located in the same plane, and the shooting target also needs to be in the same plane, and there is no requirement for the shooting distance.

[0049] Step S3, the infrared camera array simultaneously takes a sample of the cooperative target;

[0050] Specifically, the infrared camera array simultaneously takes a sample of the cooperative target (three infrared sources) to obtain the same number of images as the cameras. The infrared camera array in the same plane simultaneously samples the three infrared sources in the same plane.

[0051] The brightness of the infrared source in the collected image should be much greater than the environment, so as to convert the grayscale image into a black and white image through grayscale threshold binarization.

[0052] Referring to Figure 3 In the embodiment, in step S3, the image obtained by the infrared camera is a single-channel grayscale image, which is converted into a binary image through a grayscale threshold. The threshold should be set between the grayscale value of the cooperative target region and the grayscale value of the background region, so as to distinguish the two. In addition, due to the existence of noise points, i.e. pixel points with excessively high or low grayscale values, in the infrared focal plane array imaging inside the infrared camera, the noise points will interfere with step S4. Therefore, the noise points are removed through a morphological erosion operation, wherein the size of the erosion matrix should meet the condition of eliminating the noise points and retaining the cooperative target region.

[0053] Step S4, the collected image is binarized, and the interference noise points formed in the imaging are removed through a morphological erosion operation, which generally removes small-area noise points.

[0054] Further, according to the imaging characteristics of the infrared camera, the black and white image is subjected to erosion and expansion processing to remove the interference noise points formed in the imaging.

[0055] Step S5, the reference points are obtained by further marking the connected domain and calculating the mean value of the connected domain coordinates.

[0056] Further, according to the volume and shape characteristics of the infrared source, the reference point coordinates of the three cooperative targets are obtained by averaging the coordinate values of the connected domain of the black and white image, and the corresponding relationship between the three points in different images is obtained through the distance relationship between the three points.

[0057] In the embodiment, the collected image is converted into a black and white image through grayscale binarization, small-area noise points are removed through a morphological erosion operation, three white regions (corresponding to the three infrared sources) are obtained, and three points are obtained by further marking the connected domain and calculating the mean value of the connected domain coordinates.

[0058] In step S5, the connected domains of the three cooperative target corresponding regions in the binary image are obtained by computer graphics method, and the mean value of all pixels in each connected domain is obtained to obtain three reference points. The advantage is that the error caused by infrared source imaging divergence can be reduced by taking the mean value of all pixels in each connected domain. In addition, three coordinate points are required for the radiation transformation in step S6 instead of regions, which is the necessity of converting the connected domain into a point.

[0059] Reference point coordinate formula:

[0060]

[0061]

[0062] wherein X i , Y i are the horizontal and vertical coordinates of all pixels in the same connected domain. N is the number of pixel points in the connected domain, and X, Y are the horizontal and vertical coordinate mean values of the calculated connected domain.

[0063] Step S6, affine transformation is performed through the three points.

[0064] The present application obtains the corresponding relationship of three points in multiple images captured by the infrared camera array through the different distance relationships (distance ratios) between the three cooperative targets. Affine transformation is performed through the corresponding relationship of the three points between different images.

[0065] In step S6, the reference points are labeled through the distance ratio of 3:4:5 of the three reference points, three pairs of coordinates are required for affine transformation, one image needs to be selected as a reference image, and the corresponding relationship between the points needs to be found. Through the different distance characteristics of the three reference points, the characteristic value of each reference point in the triangle formed by the three reference points is the length of the opposite side of the point in the triangle. The reference points are sorted and labeled according to the characteristic value size, the corresponding relationship between the reference points is determined, and the reference point set of the infrared camera array affine transformation is obtained. The image captured by one camera in the infrared camera array is determined as the reference, and the images of other cameras in the infrared camera array are made to have the same spatial position through affine transformation. Affine transformation formula:

[0066]

[0067] wherein a, b, c, d, e, f are transformation parameters calculated through affine transformation, x, y represent the original horizontal and vertical coordinates, and x', y' represent the coordinates after affine transformation.

[0068] So far, through infrared source cooperation target image sampling, reference point acquisition and affine transformation, the conversion relationship of infrared camera array in cooperation target plane area shooting image is calculated, that is, affine transformation matrix. Then the final common area can be obtained by intercepting the overlapping area.

[0069] Step S7, the image overlapping part is intercepted as the final common area;

[0070] Further, the image after affine transformation is traversed by pixel, and the image overlapping part is intercepted as the final common area result.

[0071] Based on the three reference points of the image obtained by the camera array, affine transformation is carried out, and then the overlapping area is obtained, that is, the maximum common area.

[0072] Referring to Figure 4 , the form of the selected common area is a horizontal rectangle, the upper boundary of the rectangle is the minimum value of the Y coordinate of the common area point set (Y top ), the lower boundary is the maximum value of the Y coordinate of the common area point set (Y bottom ), the left boundary is the minimum value of the X coordinate of the common area point set (X left ), and the right boundary is the maximum value of the X coordinate of the common area point set (X right ). The formula is as follows:

[0073] X left = Min(x) (4)

[0074] X right = Max(x) (5)

[0075] Y top = Min(y) (6)

[0076] Y bottom = Max(y) (7)

[0077] Wherein, x, y are the set of horizontal coordinates and the set of vertical coordinates of the common area, Min() represents taking the minimum value, and Max() represents taking the maximum value.

[0078] The final result is in the form of an image transformation matrix and an external rectangle of the common area (that is, a rectangular image is returned, and the content of the image is the obtained maximum common area, which is used for motor vehicle exhaust detection concentration inversion). The image transformation matrix is the affine transformation matrix, and the number is n-1 (n is the number of cameras) because the reference image is excluded. The matrix form is four positive integers, including the left upper corner of the matrix and the horizontal and vertical coordinates and the length and width, and the number is n.

[0079] In particular implementation, the method provided by the technical scheme of the present application can be automatically run by a computer software technology, and a system device of the method, such as a computer readable storage medium storing a computer program of the technical scheme of the present application and a computer device running the computer program, should also be within the protection scope of the present application.

[0080] In some possible embodiments, a public region extraction system based on a cooperative target infrared camera array is provided, comprising a processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a public region extraction method based on a cooperative target infrared camera array as described above.

[0081] In some possible embodiments, a public region extraction system based on a cooperative target infrared camera array is provided, comprising a readable storage medium, and a computer program is stored on the readable storage medium, and the computer program is executed to implement a public region extraction method based on a cooperative target infrared camera array as described above.

[0082] The above description is only a preferred embodiment of the present application, but the protection scope of the present application is not limited to this. Any modification or replacement within the technical range disclosed by the present application can be easily thought by those skilled in the art, and should be covered within the protection scope of the present application.

[0083] The specific embodiments described herein are merely illustrative of the spirit of the present application. Those skilled in the art of the present application can make various modifications or supplements to the described specific embodiments or replace them with similar ways without departing from the spirit of the present application or exceeding the scope defined by the appended claims.

Claims

1. A method for public region extraction of an infrared camera array based on a cooperative target, characterized in that: The following processing is performed, The selection of the cooperation target includes setting the cooperation target to use infrared sources, and the wavelength band of the generated infrared light is in or covers the overlapping part of the corresponding wavelength band of the infrared camera array; The placement of the cooperation target includes installing three cooperation targets on the same plane, the distances between the three cooperation targets are not equal in the image captured by the camera array, and the distance ratios between the three cooperation targets are determined; When the cooperation target is placed, the three infrared sources are installed on the same plane and arranged in a right-angled triangle; The infrared camera array simultaneously captures and samples the three cooperation targets to obtain the same number of images as the number of cameras in the infrared camera array; The collected images are converted into black and white images through gray-scale binary conversion, and the interference noise points formed in imaging are removed through morphological erosion operation to obtain the white areas corresponding to the three cooperation targets, respectively, and then the mean values of the coordinates of the connected domains are obtained to obtain three points; The corresponding relationship of the three points in the multiple images captured by the infrared camera array is obtained through the different distances between the three cooperation targets, and affine transformation is performed on the corresponding relationship of the three points between different images; wherein the brightness of the infrared source in the collected image is much higher than the environment, and the gray-scale image is converted into a black and white image through gray-scale threshold binary conversion; according to the imaging characteristics of the infrared camera, the black and white image is subjected to erosion and expansion processing to remove the interference noise points formed in imaging; according to the volume and shape characteristics of the infrared source, the average of the coordinate values of the connected domains of the black and white image is obtained to obtain the reference point coordinates of the three cooperation targets, and the corresponding relationship between the three points in different images is obtained through the distance relationship between the three points. The image after affine transformation is traversed by pixels, and the overlapping part of the image is cut out as the final common area result.

2. The common area extraction method of the infrared camera array based on the cooperation target according to claim 1, characterized in that: When the three infrared sources are installed on the same plane and arranged in a right-angled triangle, the distance ratio between the three infrared sources is set to 3:4:

5.

3. The method of claim 2, wherein: The infrared camera array located on the same plane simultaneously samples the three infrared sources on the same plane.

4. The method of claim 1, wherein: Affine transformation is performed based on the three reference points of the image obtained by the camera array, and then the maximum common area is obtained by acquiring the overlapping area.

5. The method of claim 1 or 2 or 3 or 4, wherein: Used for motor vehicle exhaust detection concentration inversion.

6. A cooperative target based public region extraction system for an infrared camera array, comprising: The processor and the memory are included, the memory is used for storing program instructions, and the processor is used for calling the storage instructions in the memory to execute the common area extraction method of the infrared camera array based on the cooperation target according to any one of claims 1-5.

7. A cooperative target based public region extraction system for an infrared camera array, comprising: The readable storage medium is included, and the computer program is stored on the readable storage medium, and the computer program is executed to realize the common area extraction method of the infrared camera array based on the cooperation target according to any one of claims 1-5.

Images