Multi-sensor image and position attitude information synchronous transmission method

Abstract

The invention provides a multi-sensor image and position attitude information synchronous transmission method, which belongs to the technical field of embedded systems, and is characterized in that a system control unit obtains time from an upper computer through a network time protocol, and performs time synchronization with each sensor through a second pulse signal; the system control unit sends an external trigger signal to control the sensor to image, and records current time, position attitude and angle data to a circular buffer queue; when the sensor performs imaging, the exposure moment is superposed on an image, the image processing unit analyzes the exposure moment on the image and compresses the image, and then the image and the exposure moment are sent to the system control unit; and the system control unit searches corresponding position attitude and angle information from the circular buffer queue according to a nearest neighbor method, and then sends the position attitude and angle information and the image data to the upper computer. According to the invention, the synchronization of the image of each sensor and the position attitude information during imaging is ensured through the correspondence of the time information, and the influence caused by image processing and transmission delay is eliminated.

Description

Technical Field

[0001] The present application relates to the field of embedded systems, and in particular to a method for synchronously transmitting multi-sensor images and position and posture information. Background Art

[0002] Airborne photoelectric detection systems usually place imaging sensors on a rotatable servo mechanism, and control the servo mechanism to rotate in azimuth and pitch to achieve search or scanning imaging of the target area. Usually, each imaging sensor of the photoelectric detection system sends the scanned image to the system control unit, which then sends the scanned image, the position and posture of the carrier aircraft during imaging, the aiming line angle of the photoelectric detection system, etc. to the host computer. The host computer can use the position and posture information during imaging to guide subsequent image stitching and other processing processes.

[0003] However, when imaging, each sensor of the photoelectric detection system needs to go through the processes of exposure, image processing, and image output. There is a delay between the time when the image reaches the system control unit and the time when exposure starts. In addition, the delay is different for different sensors or different imaging scenarios. The time when the sensor image and position and posture information arrive at the system control unit is not synchronized. Summary of the invention

[0004] In view of this, the present application provides a method for synchronous transmission of multi-sensor images and position and posture information to solve the problem of the time asynchrony between the sensor images and position and posture information arriving at the system control unit, so that the images and position and posture information are sent to the host computer synchronously.

[0005] The present application provides a method for synchronously transmitting multi-sensor images and position and posture information using the following technical solutions:

[0006] A method for synchronously transmitting multi-sensor images and position and posture information, characterized in that it comprises the following steps:

[0007] The system control unit obtains the current time from the host computer through the network time protocol. Each sensor updates its own time by receiving the second pulse signal of the inertial navigation unit and the millisecond time sent by the system control unit through the bus. Before updating the time, the sensor first checks the synchronization between the second pulse signal and the time sent by the bus.

[0008] The system control unit controls the sensor imaging by sending an external trigger signal to each sensor at a fixed frequency;

[0009] When the system control unit sends the external trigger signal, it obtains the current time, the position and attitude data of the inertial navigation unit and the angle data of the angle measurement unit, and stores them in the circular buffer queue;

[0010] When the sensor receives an external trigger signal to start exposure imaging, it first records the current time and superimposes the exposure time on each frame of the image;

[0011] The image processing unit receives the image sent by the sensor, parses the exposure time of the current frame from the previous image, compresses the image, and then sends the compressed image and the exposure time to the system control unit;

[0012] After receiving the image and time information, the system control unit searches for the position, attitude and angle information closest to the exposure time from the circular buffer queue in a nearest neighbor manner, and then sends the image, position, attitude and angle data together to the host computer.

[0013] Optionally, the step of each sensor updating its own time includes: the inertial navigation unit sends a second pulse signal to the system control unit and each sensor once per second; after receiving the second pulse signal, the system control unit sends the current time in milliseconds to each sensor through the bus; after each sensor receives the second pulse signal, its own millisecond counter is cleared and restarts timing; after receiving the time information sent by the system control unit, the received time is added to the time of the millisecond counter as the current absolute time.

[0014] Optionally, the step of the sensor checking the synchronization between the second pulse signal and the time sent by the bus includes: the system control unit and each sensor uniformly count the received second pulse signals, the second pulse count of the system control unit and the time information are sent to the sensor through the bus, after the sensor receives the time information, the second pulse count of the system control unit is compared with its own second pulse count, if they are not equal, the received time information data is discarded, and the next second pulse is waited for to perform time synchronization again.

[0015] Optionally, the process by which the system control unit acquires the position and attitude data of the inertial navigation unit and the angle data of the angle measurement unit is as follows: the inertial navigation unit and the angle measurement unit are connected to the system control unit via an RS422 bus, and the inertial navigation unit and the angle measurement unit respectively send the position and attitude information and azimuth and pitch angle information of the photoelectric detection system to the system control unit with a period of 5 milliseconds.

[0016] Optionally, the method of superimposing the exposure moment onto each frame of image includes: replacing the original data of the first 8 pixel positions of each frame of image with the exposure moment data.

[0017] Optionally, when superimposing the time information onto the first 8 pixels of the visible light sensor image, each pixel of the color image has multiple components, and only one of the components of each pixel is replaced.

[0018] In summary, this application includes the following beneficial technical effects:

[0019] The present invention ensures the time consistency of the entire system by using the NTP protocol and the second pulse signal, and at the same time ensures the synchronization of the image of each sensor and the position and posture information during imaging through the correspondence of time information, eliminating the influence caused by the delay in the image processing and transmission process. The present invention has been successfully applied to a certain photoelectric detection equipment and has good real-time performance and reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 4 is a structural block diagram of the airborne photoelectric detection system in an embodiment of the present application.

[0022] Figure 2 This is a schematic diagram of the working process of the system control unit in the embodiment of the present application;

[0023] Figure 3 This is a schematic diagram of the working process of the sensor in the embodiment of the present application;

[0024] Figure 4 This is a schematic diagram of the workflow of the image processing unit of this application. DETAILED DESCRIPTION

[0025] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0026] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0027] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present application, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0028] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application. The illustrations only show the components related to the present application rather than being drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0029] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0030] The embodiment of the present application provides a method for synchronously transmitting multi-sensor images and position and posture information.

[0031] like Figures 1 to 4 As shown. In a specific embodiment, the airborne photoelectric detection system includes a system control unit, a visible light sensor, an infrared sensor, an image processing unit, an inertial navigation unit and an angle measurement unit. The system control unit is connected to the host computer via Ethernet for NTP time synchronization and sending information such as images and position and attitude to the host computer; the inertial navigation unit and the angle measurement unit are connected to the system control unit via the RS422 bus, and send the position and attitude information and the azimuth and pitch angle information of the photoelectric detection system to the system control unit in a period of 5 milliseconds; the inertial navigation unit sends a second pulse signal to the system control unit and each sensor once per second for system time synchronization; the system control unit is connected to the sensor and the image processing unit via the CAN bus for sending time information and control instructions; when the photoelectric detection system is in scanning imaging mode, the system control unit sends an external trigger signal to each sensor at a certain frequency; the sensor is connected to the image processing unit via the SDI interface, and the image processing unit compresses the image and transmits it to the system control unit via the PCIe interface.

[0032] A method for synchronously transmitting multi-sensor images and position and posture information comprises the following steps:

[0033] The system control unit obtains the current time from the host computer through the network time protocol. Each sensor updates its own time by receiving the second pulse signal of the inertial navigation unit and the millisecond time sent by the system control unit through the bus. Before updating the time, the sensor needs to check the synchronization between the second pulse signal and the time sent by the bus.

[0034] The system control unit controls the sensor imaging by sending external trigger signals to each sensor at a fixed frequency.

[0035] When sending the external trigger signal, the system control unit obtains the current time, the position and attitude data of the inertial navigation unit and the angle data of the angle measurement unit, and stores the current time, the position and attitude data of the inertial navigation unit and the angle data of the angle measurement unit into a circular buffer queue.

[0036] When the sensor receives an external trigger signal to start exposure imaging, it first records the current time and superimposes the exposure time on each frame of the image. The method of superimposing the exposure time on each frame of the image includes: replacing the original data of the first 8 pixel positions of each frame of the image with the exposure time data. When the sensor superimposes the time information on the first 8 pixels of each frame of the image, considering that the output of the visible light sensor is a color image, when using a coding format such as YUV or RGB, each pixel point will have multiple components. When superimposing the time, the present invention only replaces one of the components of each pixel.

[0037] The image processing unit receives the image sent by the sensor, parses the exposure time of the current frame from the previous image, compresses the image, and then sends the compressed image and the exposure time to the system control unit. After receiving the image sent by the sensor, the image processing unit parses the exposure time of the current frame from the first 8 pixels, compresses the image, and then sends the compressed image and the parsed exposure time to the system control unit.

[0038] After receiving the image and time information, the system control unit searches for the position, attitude and angle information closest to the exposure time from the circular buffer queue in a nearest neighbor manner, and then sends the image, position, attitude and angle data together to the host computer.

[0039] This application ensures the time consistency of the entire system by using the network time protocol and second pulse signal, and at the same time ensures the synchronization of the position and posture information of each sensor image and imaging through the correspondence of time information, eliminating the impact caused by delays in image processing and transmission. The present invention has been successfully applied to a certain photoelectric detection device and has good real-time performance and reliability.

[0040] The specific steps for each sensor to update its own time include: the inertial navigation unit sends a second pulse signal to the system control unit and each sensor once per second; after receiving the second pulse signal, the system control unit sends the current time in milliseconds to each sensor through the bus; after each sensor receives the second pulse signal, its own millisecond counter is cleared and restarts timing; after receiving the time information sent by the system control unit, the received time is added to the time of the millisecond counter as the current absolute time.

[0041] When the sensor receives the second pulse signal and the time sent by the system control unit through the bus, the second pulse signal and the bus time may be misaligned due to bus communication delay or failure. The steps for the sensor to check the synchronization of the second pulse signal and the time sent by the bus include: the system control unit and each sensor uniformly count the received second pulse signal, and the second pulse count of the system control unit is sent to the sensor through the bus together with the time information. After receiving the time information, the sensor compares the received system control unit second pulse count with its own second pulse count. If they are not equal, the received time information data is discarded and the next second pulse is waited for to synchronize the time again.

[0042] In the embodiment of the present application, the system control unit obtains time from the host computer through the NTP protocol based on Ethernet. The NTP protocol can minimize the impact of network delays and achieve millisecond-level or higher time synchronization accuracy through complex algorithms and multiple exchanges of time information. Among them, the full name of NTP is Network Time Protocol, which means network time protocol in Chinese.

[0043] In order to synchronize the sensor time with the system control unit time, after the system initialization is completed and each unit is working normally, the inertial navigation unit sends a second pulse signal to the system control unit and each sensor through the LVDS interface once per second. After receiving the second pulse signal, the system control unit counts the second pulses and immediately broadcasts the current time and second pulse count to each sensor through the CAN bus; the current time is the number of milliseconds from 0:00 on January 1, 1970, expressed in 6 bytes, and the second pulse count is expressed in 2 bytes.

[0044] After receiving the second pulse signal, the sensor counts the second pulses, then clears its own millisecond timer and restarts the timing. After receiving the CAN bus time information, the received second pulse count is compared with its own second pulse count. If they are equal, the received CAN bus time and the time of its own millisecond timer are added as the latest time to complete a time synchronization; if they are not equal, the judgment is made when the next second pulse and CAN bus time arrive.

[0045] When the photoelectric detection system works in scanning imaging mode, the system control unit sends an external trigger signal to the sensor at a certain frequency through the LVDS interface to control the sensor exposure imaging. While the system control unit sends the external trigger signal, it obtains the current position attitude data and azimuth pitch angle data from the inertial navigation unit and the angle measurement unit through the RS422 bus, and stores the current time, position attitude data, and angle data in the circular buffer. After receiving the external trigger signal, the sensor records the current time and then starts exposure imaging. When outputting each frame of the image, the first 8 pixel values ​​are replaced with the recorded exposure time. For the visible light sensor, the output image adopts the YUV encoding format. Each pixel has three components of Y, U, and V. When replacing the pixel value with time, only the Y component of each pixel is replaced.

[0046] The sensor outputs images to the image processing unit through the SDI interface. After receiving the image, the image processing unit first parses the exposure time of the image from the first 8 pixels, then compresses each frame of the image using the JPEG 2000 format, and then sends the exposure time of the image and the compressed image to the system control unit through the PCIe interface.

[0047] After receiving the image data, the system control unit first parses the exposure time of the image, and then searches the circular buffer for the position, attitude and angle data closest to the exposure time according to the nearest neighbor principle. Finally, the image, exposure time, position, attitude and angle data are sent to the host computer together through Ethernet in the agreed format.

[0048] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for synchronously transmitting multi-sensor images and position and posture information, characterized in that: The steps include: The system control unit obtains the current time from the host computer through the network time protocol. Each sensor updates its own time by receiving the second pulse signal of the inertial navigation unit and the millisecond time sent by the system control unit through the bus. Before updating the time, the sensor first checks the synchronization between the second pulse signal and the time sent by the bus. The system control unit controls the sensor imaging by sending an external trigger signal to each sensor at a fixed frequency; When the system control unit sends the external trigger signal, it obtains the current time, the position and attitude data of the inertial navigation unit and the angle data of the angle measurement unit, and stores them in the circular buffer queue; When the sensor receives an external trigger signal to start exposure imaging, it first records the current time and superimposes the exposure time on each frame of the image; The image processing unit receives the image sent by the sensor, parses the exposure time of the current frame from the previous image, compresses the image, and then sends the compressed image and the exposure time to the system control unit; After receiving the image and time information, the system control unit searches for the position, attitude and angle information closest to the exposure time from the circular buffer queue in a nearest neighbor manner, and then sends the image, position, attitude and angle data together to the host computer.

2. The method for synchronously transmitting multi-sensor images and position and posture information according to claim 1, characterized in that: The steps for each sensor to update its own time include: the inertial navigation unit sends a second pulse signal to the system control unit and each sensor once per second; after receiving the second pulse signal, the system control unit sends the current time in milliseconds to each sensor through the bus; after receiving the second pulse signal, each sensor resets its own millisecond counter and restarts timing; after receiving the time information sent by the system control unit, the received time is added to the time of the millisecond counter as the current absolute time.

3. The method for synchronously transmitting multi-sensor images and position and posture information according to claim 1, characterized in that: The steps of the sensor checking the synchronization between the second pulse signal and the time sent by the bus include: the system control unit and each sensor uniformly count the received second pulse signal, the second pulse count of the system control unit and the time information are sent to the sensor through the bus, after the sensor receives the time information, it compares the received system control unit second pulse count with its own second pulse count, if they are not equal, the received time information data is discarded, and the next second pulse is waited for time synchronization again.

4. The method for synchronously transmitting multi-sensor images and position and posture information according to claim 3, characterized in that: The process by which the system control unit obtains the position and attitude data of the inertial navigation unit and the angle data of the angle measurement unit is as follows: the inertial navigation unit and the angle measurement unit are connected to the system control unit via the RS422 bus, and the inertial navigation unit and the angle measurement unit send the position and attitude information and azimuth and pitch angle information of the photoelectric detection system to the system control unit at a period of 5 milliseconds.

5. The method for synchronously transmitting multi-sensor images and position and posture information according to claim 1, characterized in that: The method of superimposing the exposure time onto each frame of image includes: replacing the original data of the first 8 pixel positions of each frame of image with the exposure time data.

6. The method for synchronously transmitting multi-sensor images and position and posture information according to claim 5, characterized in that: When superimposing the time information onto the first 8 pixels of the visible light sensor image, each pixel of the color image has multiple components, and only one of the components of each pixel is replaced.

Images