Real-time multi-dimensional imaging adaptive adjustment system and method based on deep learning

Abstract

A real-time multi-dimensional imaging adaptive adjustment system and method based on deep learning, belonging to the field of optoelectronic imaging technology, in order to solve the image field difference caused by the camera acquisition orientation difference after multi-dimensional image acquisition in the prior art, the method of manually adjusting the camera orientation to reduce the error has the problems of large workload and large error that cannot be controlled, the system includes an optical imaging system, an image information processing unit, and a pan-tilt control system, wherein the optical imaging system and the image information processing unit are placed together on the pan-tilt control system; the optical imaging system and the image information processing unit are electrically connected, and the image information processing unit and the pan-tilt control system are electrically connected; the image information processing unit mainly performs image processing on the optical imaging system, obtains the coordinate offset information of the image after processing, transmits the coordinate offset information to the pan-tilt control system, and then performs the next step of system adjustment. The automatic adjustment system of the present invention can improve work efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of optoelectronic imaging technology, and in particular relates to a real-time multi-dimensional imaging adaptive adjustment system and method based on deep learning. Background Art

[0002] With the development of optoelectronic imaging technology, the single-dimensional imaging mode has gradually developed into a multi-dimensional mode. Due to the limitations of the detector's own physical properties, imaging mechanism, and observation angle, a single two-dimensional detector often cannot extract enough feature points from a multi-dimensional scene. A single two-dimensional detector cannot independently obtain a multi-dimensional image of a target in a scene, and multi-source image fusion technology is required. Image fusion technology is to fuse the image feature information captured by sensors of different bands into a new image through algorithm processing. The matching and fusion of multi-dimensional images can highlight the feature points of different dimensions and improve the image contrast.

[0003] Image registration is the basis of image fusion. Image registration is the process of matching and superimposing two or more images acquired at different times, with different sensors (imaging devices) or under different conditions (weather, illumination, camera position and angle, etc.) by finding the spatial mapping relationship between the pixels of one image and the pixels of another image.

[0004] However, in the process of multi-dimensional image acquisition, there will be differences in camera shooting angles and camera orientations, which will lead to nonlinear offsets in the images after registration and fusion. The field of view of images captured by different cameras will also be different, resulting in a reduction in the effective range after matching and fusion. The traditional method is to use manual adjustment of the camera or a three-pull and three-tighten device to calibrate the camera imaging error, but there are problems such as large workload and uncontrollable errors. At present, there is no system that can acquire multi-dimensional images in real time and automatically adjust the camera orientation and imaging field of view. Therefore, there is an urgent need for a system that can calibrate the acquired multi-dimensional images and the offset of the acquired images in real time. Summary of the invention

[0005] In order to solve the problem of image field difference caused by camera acquisition orientation difference after multi-dimensional image acquisition in the prior art, the method of manually adjusting the camera orientation to reduce the error has the problems of large workload and large and uncontrollable error. This invention proposes a real-time multi-dimensional imaging adaptive adjustment system and method based on deep learning, which uses the convolutional neural network in deep learning to extract features of images of different dimensions. The trained network can quickly and accurately obtain the geometric consistency error between different images, that is, the translation amount of the image in the horizontal and vertical directions, and transmit the obtained offset to the control system. The controller is used to adjust the camera system in real time, which can quickly and automatically adjust the camera orientation to make the image field of view obtained by cameras with different spectra consistent, so as to ensure the image consistency of subsequent image registration and image fusion. The fused image has higher accuracy and contains richer image information. At the same time, the automatic adjustment system can greatly reduce labor costs and improve work efficiency.

[0006] A real-time multi-dimensional imaging adaptive adjustment system based on deep learning is characterized in that the system includes an optical imaging system, an image information processing unit, and a pan-tilt control system, wherein the optical imaging system and the image information processing unit are placed together on the pan-tilt control system; the optical imaging system and the image information processing unit are electrically connected, and the image information processing unit and the pan-tilt control system are electrically connected; the image information processing unit mainly performs image processing on the optical imaging system, obtains coordinate offset information of the image after processing, transmits the coordinate offset information to the pan-tilt control system, and then performs the next step of system adjustment.

[0007] The optical imaging system comprises an infrared imaging camera and a visible light imaging camera; the infrared imaging camera and the visible light imaging camera are on the same horizontal line and image the same target object.

[0008] The image information processing unit includes an image preprocessing module, a convolutional neural network, a loss function and a coordinate offset information module. The image preprocessing module receives the image formed by the optical imaging system and performs preprocessing, namely, mean filtering and image scale transformation. The image preprocessing module inputs the processed image information into the convolutional neural network to obtain image contour information and geometric consistency loss, and then stores the image contour information and geometric consistency loss information in the loss function. The loss function obtains the coordinate offset information through calculation, and then stores the coordinate offset information in the coordinate offset information module.

[0009] The pan / tilt control system includes a stepper motor, an optical system clamp, a turntable, a stabilizing platform and a control system. The front opening of the stabilizing platform is used to fix the front end of the optical imaging system. The rear of the stabilizing platform is the turntable and the optical system clamp. The optical system clamp fixes the rear end of the optical imaging system to the turntable. The turntable can be adjusted left and right and in pitch, so that the front end of the optical imaging system is stabilized and the rear end is adjusted, thereby finally realizing the control of the optical system.

[0010] A real-time multi-dimensional imaging adaptive adjustment method based on deep learning, characterized in that the method comprises the following steps:

[0011] Step 1, obtaining multi-source images;

[0012] In the optical imaging system, the infrared imaging camera and the visible light imaging camera respectively image the remote target. First, the focal length of the infrared imaging camera is fixed, and then the central target is determined to obtain the infrared image. Then, the focal length of the visible light imaging camera is adjusted so that the central target of the optical image is consistent with the central target of the infrared image, and the field of view of the visible light image is consistent with the field of view of the infrared image. Then, the visible light field of view is adjusted in real time by the stepper motor.

[0013] Step 2: Obtain multi-source image coordinate offset information;

[0014] After the optical imaging system acquires the infrared image and the visible light image, the image preprocessing module first performs mean filtering on the image and performs image scale transformation. The transformed images are of the same size. Then, the preprocessed infrared image and visible light image are used as the input of the convolutional neural network. The convolutional neural network first performs semantic segmentation on the visible light image and the infrared image, obtains the contour information of the infrared image and the visible light image, and compares the rough offset of the image contour. Then, the convolutional neural network extracts the features of the two images, obtains the image features, compares each pixel, and calculates the geometric consistency error value, that is, the rigid offset of the image in the horizontal and vertical directions. The obtained contour information and geometric consistency error are stored in the loss function. The loss function calculates and feeds back the offset information to the coordinate offset information module, and obtains the rough offset and the rigid offset in the horizontal and vertical directions. Thus, the coordinate offset information of the multi-source image is acquired.

[0015] Step 3: Adjust the control pan / tilt;

[0016] The control system receives the rough image offset information and the pixel offset in the horizontal and vertical directions. The control system processes the rough image offset data, then controls and adjusts the stepper motor voltage pulse, controls the stepper motor for coarse adjustment, and then transmits the pixel offset in the horizontal and vertical directions to the stepper motor for fine adjustment. The stepper motor drives the turntable to move left and right and in pitch, thereby controlling the movement of the visible light imaging camera to achieve the correction of the field of view range of the camera imaging.

[0017] Beneficial effects of the present invention:

[0018] 1. The preprocessed image of the present invention is input into a trained convolutional neural network to obtain the offset information of the two images in real time, and the feedback offset information is transmitted to the control system to achieve real-time adjustment of the camera orientation;

[0019] 2. The present invention acquires images with consistent fields of view in real time, reduces nonlinear offset, improves the accuracy of subsequent image registration and fusion images, meets the accuracy requirements of image registration and fusion, and can expand the matching range and acquire richer target image information;

[0020] 3. The system is highly adaptable and can calibrate cameras of different dimensions;

[0021] 4. Compared with the traditional manual adjustment system, the practical automatic adjustment method of the present invention shortens the adjustment time and saves labor costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a structural schematic diagram of the real-time multi-dimensional imaging adaptive adjustment system based on deep learning of the present invention.

[0023] Figure 2 This is the structural diagram of the pan-tilt control system.

[0024] Figure 3 The present invention is a flowchart of a process for an image processing unit to obtain coordinate offset information for multi-source image processing.

[0025] Figure 4 Flowchart of the motor adjustment process controlled according to image offset.

[0026] in Figure 1 1 is the optical imaging system, 2 is the image information processing unit, 3 is the PTZ control system, 4 is the target, 1-1 is the infrared imaging camera, 1-2 is the visible light imaging camera, 2-1 is the image preprocessing module, 2-2 is the convolutional neural network, 2-3 is the loss function, and 2-4 is the coordinate offset information module. Figure 2 Among them, 3-1 is a stepper motor, 3-2 is an optical system clamp, 3-3 is a turntable, 3-4 is a stabilizing platform, and 3-5 is a control system. DETAILED DESCRIPTION

[0027] The present invention is further described below in conjunction with the accompanying drawings.

[0028] like Figure 1 As shown, a real-time multi-dimensional imaging adaptive adjustment system based on deep learning includes an optical imaging system 1, an image information processing unit 2, and a pan-tilt control system 3, wherein the optical imaging system 1 and the image information processing unit 2 are placed together on the pan-tilt control system 3; the optical imaging system 1 and the image information processing unit 2 are electrically connected, and the image information processing unit 2 and the pan-tilt control system 3 are electrically connected; the image information processing unit 2 mainly performs image processing on the optical imaging system 1, obtains the coordinate offset information of the image after processing, transmits the coordinate offset information to the pan-tilt control system 3, and then performs the next step of system adjustment.

[0029] The optical imaging system 1 includes an infrared imaging camera 1-1 and a visible light imaging camera 1-2. The infrared imaging camera 1-1 and the visible light imaging camera 1-2 are at the same level and image the same target object.

[0030] The infrared imaging camera 1-1 includes a short-wave infrared camera and a short-wave infrared zoom lens. The short-wave infrared detector is a GH-SCS00 short-wave infrared camera produced by Shanxi Guohui Co., Ltd. The detector used in the camera is a 15um 640×512 detector, and the short-wave infrared zoom lens is a GZ10X16M-MP SWIR short-wave zoom lens of Japan TAWOV, with a zoom range of 16 to 160 mm. The visible light imaging camera 1-2 is a high-definition zoom lens integrated network camera of Shenzhen Qianlimu Technology Co., Ltd., model JZC-N83020, with a zoom range of 100 to 1000 mm and a camera resolution of 1920×1080.

[0031] The image information processing unit 2 includes an image preprocessing module 2-1, a convolutional neural network 2-2, a loss function 2-3, and a coordinate offset information module 2-4; the image preprocessing module 2-1 is mainly responsible for the preprocessing of the image input, mainly the mean filter, and performs image scale transformation. The image preprocessing module 2-1 inputs the image contour information and geometric consistency loss obtained by the convolutional neural network 2-2, and the image contour information and geometric consistency loss information are stored in the loss function 2-3. The loss function 2-3 obtains the coordinate offset information module 2-4 through calculation. The image processing unit mainly performs image processing on the optical imaging system 1, obtains the coordinate offset information of the image after processing, transmits the coordinate offset information to the pan / tilt control system 3, and then performs the next step of system adjustment.

[0032] The structure of the pan / tilt control system 3 includes a stepper motor 3-1, an optical system support 3-2, a turntable 3-3, a stable platform 3-4 and a control system 3-5, wherein the control system board of the control system 3-5 is an Asus thin mini-ITX motherboard, model Pro H610TD4-CSM. The front opening of the stable platform 3-4 is used to fix the front end of the optical imaging system 1, and the rear of the stable platform 3-4 is a turntable and an optical system clamp 3-2. The optical system clamp 3-2 fixes the rear end of the optical system with the turntable 3-3. The turntable 3-3 can be adjusted left and right and in pitch, so that the front end of the optical imaging system 1 is stable and the rear end is adjusted, and finally the control of the optical imaging system 1 is realized.

[0033] Attached Figure 3 The invention is a process for the image processing unit to obtain coordinate offset information for multi-source processing. The infrared image and the visible light image formed by the infrared imaging camera 1-1 and the visible light imaging camera 1-2 are preprocessed by the image preprocessing module 2-1. The image preprocessing module 2-1 performs mean filtering on the image and performs image scale transformation. The infrared image and the visible light image after preprocessing are used as inputs of the convolutional neural network. The neural network extracts feature information of the two images respectively, performs semantic segmentation on the two images to obtain semantic segmentation information, i.e., image contour information, compares the contours to obtain a rough offset, and then uses the feature information to compare pixels of the two images to obtain a geometric consistency loss function of the two images, calculates the geometric consistency error value, i.e., the pixel offset of the image in the horizontal and vertical directions, and the loss function feeds back the offset information to the coordinate offset information module. The rough offset and the pixel offset in the horizontal and vertical directions determine the amount of subsequent camera adjustment.

[0034] Attached Figure 4 This is the process of motor adjustment after image offset information is acquired. After the image information is acquired, the offset of the two images is acquired by the image processing unit 2. The conversion of the offset is called a control instruction. After that, the control system 3-5 receives the instruction and controls the stepper motor 3-1 to adjust, so as to adjust the camera imaging range. After that, the image acquisition and adjustment process is repeated to realize the field of view alignment of multi-source images.

[0035] A real-time multi-dimensional imaging adaptive adjustment method based on deep learning, the method comprises the following steps:

[0036] Step 1: Obtain multiple source images.

[0037] In the optical imaging system 1, the infrared imaging camera 1-1 and the visible light imaging camera 1-2 respectively image the remote target 4. First, the focal length of the infrared imaging camera 1-1 is fixed, and then the central target is determined to obtain an infrared image. Then, the focal length of the visible light imaging camera 1-2 is adjusted so that the central target of the optical image is consistent with the central target of the infrared image, and the field of view of the visible light image is roughly consistent with the field of view of the infrared image. Subsequently, the visible light field of view is adjusted by the stepper motor 3-1.

[0038] Step 2: Obtain multi-source image coordinate offset information.

[0039] After the optical imaging system 1 acquires the infrared image and the visible light image, it firstly performs mean filtering on the image through the image preprocessing module 2-1, and performs image scale transformation. The sizes of the transformed images are consistent. Then, the preprocessed infrared image and visible light image are used as the input of the convolutional neural network 2-2. The convolutional neural network 2-2 first performs semantic segmentation on the visible light image and the infrared image, obtains the contour information of the infrared image and the visible light image, and compares the rough offset of the image contour. Then, the convolutional neural network 2-2 extracts the features of the two images, obtains the image features, compares each pixel, and calculates the geometric consistency error value, that is, the rigid offset of the image in the horizontal and vertical directions. The obtained contour information and geometric consistency error are stored in the loss function 2-3. The loss function 2-3 feeds back the offset information to the coordinate offset information module 2-5, and the rough offset and the rigid offset in the horizontal and vertical directions are obtained. Thus, the coordinate offset information of the multi-source image is acquired.

[0040] Step 3: Adjust the control pan / tilt.

[0041] The control system 3-5 receives the rough image offset information and the pixel offset in the horizontal and vertical directions. The control system 3-5 processes the rough image offset data, and then controls the voltage pulse of the stepper motor 3-1 to control the stepper motor 3-1 for coarse adjustment, and then transmits the pixel offset in the horizontal and vertical directions to the stepper motor 3-1 for fine adjustment. The stepper motor 3-1 drives the turntable 3-3 to move left and right and pitch, thereby controlling the movement of the visible light imaging camera to achieve the correction of the field of view of the camera imaging. The adjustment process is a real-time information acquisition and adjustment process. Since there are certain errors in the adjustment process, it needs to be adjusted repeatedly, and finally the central target of the visible light image and the infrared image are consistent, and the field of view is consistent.

Claims

1. A real-time multi-dimensional imaging adaptive adjustment system based on deep learning, characterized by: The system comprises an optical imaging system (1), an image information processing unit (2), and a pan-tilt control system (3), wherein the optical imaging system (1) and the image information processing unit (2) are placed together on the pan-tilt control system (3); the optical imaging system (1) and the image information processing unit (2) are electrically connected, and the image information processing unit (2) and the pan-tilt control system (3) are electrically connected; the image information processing unit (2) mainly performs image processing on the optical imaging system (1), obtains coordinate offset information of the image after processing, transmits the coordinate offset information to the pan-tilt control system (3), and then performs the next step of system adjustment; The optical imaging system (1) comprises an infrared imaging camera (1-1) and a visible light imaging camera (1-2); the infrared imaging camera (1-1) and the visible light imaging camera (1-2) are on the same horizontal line and image the same target object (4); The image information processing unit (2) comprises an image preprocessing module (2-1), a convolutional neural network (2-2), a loss function (2-3) and a coordinate offset information module (2-4). The image preprocessing module (2-1) receives an image formed by the optical imaging system (1) and performs preprocessing, namely, mean filtering and image scale transformation. The image preprocessing module (2-1) inputs the processed image information into the convolutional neural network (2-2) to obtain image contour information and geometric consistency loss, and then stores the image contour information and geometric consistency loss information in the loss function (2-3). The loss function (2-3) obtains coordinate offset information by calculation, and then stores the coordinate offset information in the coordinate offset information module (2-4). The pan / tilt control system (3) comprises a stepper motor (3-1), an optical system clamp (3-2), a turntable (3-3), a stabilizing platform (3-4) and a control system (3-5). The front opening of the stabilizing platform (3-4) is used to fix the front end of the optical imaging system (1). The rear of the stabilizing platform (3-4) is the turntable (3-3) and the optical system clamp (3-2). The optical system clamp (3-2) fixes the rear end of the optical imaging system (1) and the turntable (3-3). The turntable (3-3) can be adjusted left and right and in pitch, so that the front end of the optical imaging system (1) is stabilized and the rear end is adjusted, thereby finally realizing the control of the optical system.

2. A real-time multi-dimensional imaging adaptive adjustment method based on deep learning, characterized in that: The method comprises the following steps: Step 1, obtaining multi-source images; In the optical imaging system (1), an infrared imaging camera (1-1) and a visible light imaging camera (1-2) are used to image a remote target object (4) respectively. First, the focal length of the infrared imaging camera (1-1) is fixed, and then the central target object is determined to obtain an infrared image. Then, the focal length of the visible light imaging camera (1-2) is adjusted so that the central target of the optical image is consistent with the central target of the infrared image, and the field of view of the visible light image is consistent with the field of view of the infrared image. Then, the field of view of the visible light is adjusted by a stepping motor (3-1). Step 2: Obtain multi-source image coordinate offset information; After the optical imaging system (1) acquires the infrared image and the visible light image, the image preprocessing module (2-1) first performs mean filtering on the image and performs image scale transformation. The transformed images are of the same size. The preprocessed infrared image and visible light image are then used as inputs of the convolutional neural network (2-2). The convolutional neural network (2-2) first performs semantic segmentation on the visible light image and the infrared image, acquires contour information of the infrared image and the visible light image, and compares the rough offset of the image contours. The convolutional neural network (2-2) then extracts features of the two images, acquires image features, compares each pixel, and calculates the geometric consistency error value, i.e., the rigid offset of the image in the horizontal and vertical directions. The acquired contour information and geometric consistency error are stored in the loss function (2-3). The loss function (2-3) calculates and feeds back the offset information to the coordinate offset information module (2-4). The rough offset and the rigid offset in the horizontal and vertical directions are acquired. Thus, the coordinate offset information of the multi-source image is acquired. Step 3: Adjust the control pan / tilt; The control system (3-5) receives the rough image offset information and the pixel offsets in the horizontal and vertical directions, and processes the rough image offset data, and then controls and adjusts the voltage pulse of the stepper motor (3-1), controls the stepper motor (3-1) to perform rough adjustment, and then transmits the pixel offsets in the horizontal and vertical directions to the stepper motor (3-1) for fine adjustment. The stepper motor (3-1) drives the turntable (3-3) to move left and right and in pitch, thereby controlling the movement of the visible light imaging camera to achieve correction of the field of view range of the camera imaging.

Images