A multi-sensor image and position and posture information synchronous transmission method

Abstract

The application provides a multi-sensor image and position attitude information synchronous transmission method, and belongs to the technical field of embedded systems. A system control unit acquires time from an upper computer through a network time protocol, and performs time synchronization with each sensor through a second pulse signal. The system control unit sends an external trigger signal to control sensor imaging, and records current time, position attitude and angle data to a circular buffer queue. When the sensor is imaging, the exposure time is superimposed on the image. An image processing unit parses the exposure time from the image and compresses the image, and then sends the image and the exposure time to the system control unit. The system control unit searches for corresponding position attitude and angle information from the circular buffer queue according to the nearest neighbor method, and then sends the information to the upper computer together with the image data. The application ensures the synchronization of the images of each sensor and the position attitude information at the time of imaging through the corresponding time information, and eliminates the influence caused by image processing and transmission delay.

Description

Technical Field

[0001] This application relates to the field of embedded systems, and in particular to a method for synchronous transmission of multi-sensor images and position and attitude information. Background Technology

[0002] Airborne optoelectronic detection systems typically mount imaging sensors on rotatable servo mechanisms. By controlling the servo mechanism to rotate in both azimuth and pitch directions, the system can search for or scan and image a target area. Typically, the various imaging sensors in the optoelectronic detection system send the scanned images to the system control unit. The system control unit then sends the scanned images, along with the aircraft's position and attitude during imaging, and the aiming line angle of the optoelectronic detection system, to a host computer. The host computer can use this position and attitude information to guide subsequent image stitching and other processing steps.

[0003] However, the imaging process of each sensor in the photoelectric detection system involves exposure, image processing, and image output. There is a delay between the time it takes for the image to reach the system control unit and the time it takes for the exposure to begin. Moreover, the delay varies depending on the sensor or the imaging scenario. As a result, the sensor image and position and attitude information arrive at the system control unit at different times. Summary of the Invention

[0004] In view of this, this application provides a method for synchronous transmission of multi-sensor images and position and attitude information, which solves the problem of asynchronous arrival time of sensor images and position and attitude information at the system control unit, and enables the images and position and attitude information to be sent to the host computer synchronously.

[0005] This application provides a method for synchronously transmitting multi-sensor images and position / attitude information, which employs the following technical solution:

[0006] A method for synchronously transmitting multi-sensor images and position / attitude information, characterized by comprising the following steps:

[0007] The system control unit obtains the current time from the host computer via the network time protocol. Each sensor updates its own time by receiving the second pulse signal from the inertial navigation unit and the millisecond time sent by the system control unit via the bus. Before updating the time, the sensor checks the synchronization between the second pulse signal and the time sent by the bus.

[0008] The system control unit controls sensor imaging by sending external trigger signals to each sensor at a fixed frequency.

[0009] When the system control unit sends an external trigger signal, it acquires the current time, the position and attitude data of the inertial navigation unit, and the angle data of the angle measurement unit, and stores them in a circular buffer queue.

[0010] When the sensor receives an external trigger signal and begins exposure imaging, it first records the current time and then superimposes the exposure time onto each frame of the image;

[0011] The image processing unit receives the image sent by the sensor, parses the exposure time of the current frame from the previous image, compresses the image, and then sends the compressed image and the exposure time together to the system control unit.

[0012] After receiving the image and time information, the system control unit searches for the position, attitude and angle information closest to the exposure time in the circular buffer queue according to the nearest neighbor method, and then sends the image, position, attitude and angle data together to the host computer.

[0013] Optionally, the steps for each sensor to update its own time include: the inertial navigation unit sends a second pulse signal to the system control unit and each sensor every second; after receiving the second pulse signal, the system control unit sends the current time in milliseconds to each sensor via the bus; after receiving the second pulse signal, each sensor clears its own millisecond counter and restarts timing; after receiving the time information sent by the system control unit, it adds the received time to the time of the millisecond counter to obtain the current absolute time.

[0014] Optionally, the steps for the sensor to check the synchronization between the second pulse signal and the time sent by the bus include: the system control unit and each sensor count the received second pulse signal in a unified manner; the second pulse count of the system control unit is sent to the sensor along with the time information through the bus; after receiving the time information, the sensor compares the received second pulse count of the system control unit with its own second pulse count; if they are not equal, the received time information data is discarded, and the sensor waits for the next second pulse to synchronize the time.

[0015] Optionally, the process by which the system control unit acquires the position and attitude data of the inertial navigation unit and the angle data of the angle measurement unit is as follows: the inertial navigation unit and the angle measurement unit are connected to the system control unit via an RS422 bus, and the inertial navigation unit and the angle measurement unit send the position and attitude information of the photoelectric detection system and the azimuth and pitch angle information to the system control unit at a period of 5 milliseconds.

[0016] Optionally, the method of overlaying the exposure time onto each frame image includes replacing the original data at the first 8 pixel positions of each frame image with the exposure time data.

[0017] Optionally, when overlaying time information onto the first 8 pixels of the visible light sensor image, each pixel in the color image has multiple components, and only one component of each pixel is replaced.

[0018] In summary, this application includes the following beneficial technical effects:

[0019] This invention ensures the time consistency of the entire system by using the NTP protocol and a second pulse signal. Simultaneously, the correspondence of time information guarantees the synchronization of each sensor image with the position and attitude information during imaging, eliminating the impact of delays in image processing and transmission. This invention has been successfully applied to a photoelectric detection device, demonstrating excellent real-time performance and reliability. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a structural block diagram of the airborne photoelectric detection system in the embodiments of this application.

[0022] Figure 2 This is a schematic diagram of the workflow of the system control unit in the embodiments of this application;

[0023] Figure 3 This is a schematic diagram of the sensor's workflow in the embodiments of this application;

[0024] Figure 4 This is a schematic diagram of the workflow of the image processing unit in this application. Detailed Implementation

[0025] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0026] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. This application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this application, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0028] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. The illustrations only show the components related to this application and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0029] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0030] This application provides a method for synchronous transmission of multi-sensor images and position and attitude information.

[0031] like Figures 1 to 4 As shown. In a specific embodiment, the airborne optoelectronic detection system includes a system control unit, a visible light sensor, an infrared sensor, an image processing unit, an inertial navigation unit, and an angle measurement unit. The system control unit is connected to a host computer via Ethernet for NTP time synchronization and sending image and position / attitude information to the host computer. The inertial navigation unit and the angle measurement unit are connected to the system control unit via an RS422 bus, sending the position / attitude information and azimuth and pitch angle information of the optoelectronic detection system to the system control unit at 5-millisecond intervals. The inertial navigation unit sends a second pulse signal once per second to the system control unit and each sensor for system time synchronization. The system control unit is connected to the sensors and the image processing unit via a CAN bus for sending time information and control commands. When the optoelectronic detection system is in scanning imaging mode, the system control unit sends external trigger signals to each sensor at a certain frequency. The sensors are connected to the image processing unit via an SDI interface, and the image processing unit compresses the image and transmits it to the system control unit via a PCIe interface.

[0032] A method for synchronously transmitting multi-sensor images and position / attitude information includes the following steps:

[0033] The system control unit obtains the current time from the host computer via the network time protocol. Each sensor updates its own time by receiving the second pulse signal from the inertial navigation unit and the millisecond time sent by the system control unit via the bus. Before updating the time, the sensor needs to check the synchronization between the second pulse signal and the time sent by the bus.

[0034] The system control unit controls the sensor imaging by sending external trigger signals to each sensor at a fixed frequency.

[0035] When the system control unit sends an external trigger signal, it acquires the current time, the position and attitude data of the inertial navigation unit, and the angle data of the angle measurement unit, and stores the current time, the position and attitude data of the inertial navigation unit, and the angle data of the angle measurement unit into a circular buffer queue.

[0036] When the sensor receives an external trigger signal to begin exposure imaging, it first records the current time and then superimposes the exposure time onto each frame of the image. The method for superimposing the exposure time onto each frame includes replacing the original data at the first 8 pixel positions of each frame with the exposure time data. Considering that the visible light sensor outputs a color image, and each pixel will have multiple components when using encoding formats such as YUV or RGB, this invention only replaces one component of each pixel when superimposing the time information.

[0037] The image processing unit receives the image sent by the sensor, parses the exposure time of the current frame from the previous image, compresses the image, and then sends the compressed image and the exposure time together to the system control unit. Alternatively, after receiving the image from the sensor, the image processing unit parses the exposure time of the current frame from the first 8 pixels, compresses the image, and then sends the compressed image and the parsed exposure time together to the system control unit.

[0038] After receiving the image and time information, the system control unit searches for the position, attitude and angle information closest to the exposure time in the circular buffer queue according to the nearest neighbor method, and then sends the image, position, attitude and angle data together to the host computer.

[0039] This application ensures the time consistency of the entire system by using a network time protocol and a second pulse signal. Simultaneously, the correspondence of time information guarantees the synchronization of each sensor image with the position and attitude information during imaging, eliminating the impact of delays in image processing and transmission. This invention has been successfully applied to a photoelectric detection device, exhibiting excellent real-time performance and reliability.

[0040] The specific steps for each sensor to update its own time include: the inertial navigation unit sends a second pulse signal to the system control unit and each sensor every second; after receiving the second pulse signal, the system control unit sends the current time in milliseconds to each sensor via the bus; after receiving the second pulse signal, each sensor clears its own millisecond counter and restarts the timing; after receiving the time information sent by the system control unit, it adds the received time to the time of the millisecond counter to obtain the current absolute time.

[0041] When the sensor receives the second pulse signal and the time sent by the system control unit via the bus, the second pulse signal may be out of sync with the bus time due to bus communication delay or failure. The steps for the sensor to check the synchronization between the second pulse signal and the time sent by the bus include: the system control unit and each sensor count the received second pulse signal in a unified manner. The second pulse count of the system control unit is sent to the sensor along with the time information via the bus. After receiving the time information, the sensor compares the second pulse count received by the system control unit with its own second pulse count. If they are not equal, the received time information data is discarded and the sensor waits for the next second pulse to synchronize the time.

[0042] In this embodiment, the system control unit obtains the time from the host computer via Ethernet using the NTP protocol. The NTP protocol, through complex algorithms and multiple exchanges of time information, can minimize the impact of network latency and achieve time synchronization accuracy at the millisecond level or higher. NTP stands for Network Time Protocol.

[0043] To synchronize sensor time with system control unit time, after system initialization is complete and all units are operating normally, the inertial navigation unit sends a second pulse signal once per second to the system control unit and each sensor via the LVDS interface. Upon receiving the second pulse signal, the system control unit counts the second pulses and immediately broadcasts the current time and second pulse count to each sensor via the CAN bus; the current time is the number of milliseconds since 00:00 on January 1, 1970, represented by 6 bytes, and the second pulse count is represented by 2 bytes.

[0044] After receiving the second pulse signal, the sensor counts the second pulses, then clears its own millisecond timer and restarts the count. When it receives CAN bus time information, it compares the received second pulse count with its own second pulse count. If they are equal, it adds the received CAN bus time to its own millisecond timer time to obtain the latest time, completing one time synchronization. If they are not equal, it waits for the next second pulse and CAN bus time to arrive before making a judgment.

[0045] When the photoelectric detection system operates in scanning imaging mode, the system control unit sends external trigger signals to the sensor at a certain frequency via the LVDS interface to control the sensor's exposure imaging. Simultaneously, the system control unit acquires current position and attitude data and azimuth and pitch angle data from the inertial navigation unit and angle measurement unit respectively via the RS422 bus, storing the current time, position and attitude data, and angle data together in a circular buffer. Upon receiving the external trigger signal, the sensor records the current time and then begins exposure imaging. When outputting each frame, the recorded exposure time replaces the first eight pixel values. For visible light sensors, the output image uses a YUV encoding format, with each pixel having three components: Y, U, and V. When replacing pixel values ​​with time, only the Y component of each pixel is replaced.

[0046] The sensor outputs an image to the image processing unit via the SDI interface. After receiving the image, the image processing unit first parses the exposure time of the image from the first 8 pixels, then compresses each frame of the image using the JPEG 2000 format, and then sends the exposure time of the image and the compressed image together to the system control unit via the PCIe interface.

[0047] After receiving the image data, the system control unit first parses the exposure time of the image, then searches for the position, attitude and angle data closest to the exposure time in the circular buffer according to the nearest neighbor principle, and finally sends the image, exposure time, position, attitude and angle data together to the host computer via Ethernet in the agreed format.

[0048] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for synchronously transmitting multi-sensor image and position and attitude information, characterized in that, The method comprises the following steps: The system control unit obtains the current time from the host computer through the network time protocol, each sensor updates its own time by receiving the second pulse signal from the inertial navigation unit and the millisecond time sent by the system control unit through the bus, and the sensor checks the synchronization of the second pulse signal and the time sent by the bus before updating the time; The system control unit controls the imaging of the sensor by sending an external trigger signal to the sensor at a fixed frequency; When the system control unit sends the external trigger signal, it obtains the current time, the position and attitude data of the inertial navigation unit and the angle data of the angle measurement unit, and stores them in a circular buffer queue; When the sensor receives the external trigger signal to start exposure and imaging, it first records the current time, and superimposes the exposure time on each frame of image; The image processing unit receives the image sent by the sensor, parses the exposure time of the current frame from the previous image, compresses the image, and then sends the compressed image and the exposure time to the system control unit; After receiving the image and time information, the system control unit searches for the position and attitude and angle information closest to the exposure time from the circular buffer queue in a nearest neighbor manner, and then sends the image, position and attitude and angle data to the host computer; The method of superimposing the exposure time on each frame of image comprises: replacing the original data of the first 8 pixel positions of each frame of image with the exposure time data; When superimposing the time information on the first 8 pixels of the visible light sensor image, each pixel of the color image has multiple components, and only one component of each pixel is replaced.

2. The multi-sensor image and position attitude information synchronous transmission method of claim 1, wherein, The step of updating the time of each sensor comprises: the inertial navigation unit sends a second pulse signal to the system control unit and each sensor once per second, the system control unit sends the current time in milliseconds to each sensor through the bus after receiving the second pulse signal, and each sensor clears and restarts the millisecond counter after receiving the second pulse signal, and adds the received time to the time of the millisecond counter as the current absolute time after receiving the time information sent by the system control unit.

3. The method of claim 1, wherein, The step of checking the synchronization of the second pulse signal and the time sent by the bus by the sensor comprises: the system control unit and each sensor count the received second pulse signal, the system control unit sends the second pulse count to the sensor through the bus together with the time information, the sensor compares the received second pulse count of the system control unit with its own second pulse count after receiving the time information, and discards the received time information data if they are not equal, and waits for the next second pulse to synchronize the time.

4. The multi-sensor image and position attitude information synchronous transmission method of claim 3, wherein, The process of obtaining the position and attitude data of the inertial navigation unit and the angle data of the angle measurement unit by the system control unit is as follows: the inertial navigation unit and the angle measurement unit are connected to the system control unit through the RS422 bus, and the inertial navigation unit and the angle measurement unit send the position and attitude information of the photoelectric detection system and the azimuth and pitch angle information to the system control unit at a period of 5 milliseconds, respectively.

Images