Kernel mode and user mode data processing method for integrated single-photon imaging system

Abstract

A kernel mode and user mode data processing method for an integrated single-photon imaging system belongs to the technical field of optical imaging, optical signals are modulated through rapid tilt of a tiny mirror surface in a digital micromirror device, meanwhile, high-resolution image display is achieved in cooperation with an array type single-photon detector, a master controller carries a Linux operating system, and a kernel mode and a user mode are integrated. And the processor is connected with the digital micro-mirror device and the array type single-photon detector and is used for sending control and synchronization signals. The method comprises the steps that a preset mask pattern is collected in a user mode, the mask pattern is projected according to preset projection parameters, collected data and a synchronizing signal of an array type single-photon detector are transmitted to a kernel mode, a single channel or multiple channels are selected for processing according to the kernel space occupancy rate, and the kernel space occupancy rate of the array type single-photon detector is obtained. And photon data and specific synchronization data during mask projection are processed in a user mode, so that efficient data reconstruction and imaging are realized. According to the invention, the problem of real-time processing of collected special data by the single-photon laser radar system is solved, the integration and reliability of the system can be enhanced, and the application field of the system is expanded.

Description

Technical Field

[0001] The present invention belongs to the technical field of optical imaging, and relates to a kernel state and user state data processing method for an integrated single-photon imaging system. Background Art

[0002] As a mainstream optical imaging system, LiDAR is widely used in fields such as exploration, measurement and emergency rescue. However, when traditional LiDAR detects at long distances, the backscatter signal intensity decays rapidly. Especially in natural disasters and extreme weather conditions, the echo signal decays more seriously due to the influence of dust and particles in the air, resulting in insufficient detection capability of weak targets. In recent years, the rapid development of single-photon detectors has provided new opportunities for improving the performance of LiDAR systems. Single-photon LiDAR can respond to signals at the photon level, has high detection sensitivity and high temporal resolution, and can effectively detect weak signals at long distances.

[0003] However, most existing single-photon laser radar data acquisition systems use general-purpose data acquisition cards and are highly dependent on host computer software, which limits the system's integration. The ultra-long-range single-photon laser radar developed by the University of Science and Technology of China uses InGaAs / InP single-photon avalanche diodes (SPADs) as photon detection units, cooperates with general acquisition cards to obtain data, and uses external devices to realize the overall function of the system (Optica, 2021, 8(3): 344); A scanning single-pixel imaging laser radar developed by the Anhui Institute of Optics and Precision Mechanics of the Chinese Academy of Sciences uses a host computer to control digital micromirror devices for optical signal modulation, and detects the recovered photon signals through avalanche photodiodes (APDs), thereby improving the imaging resolution (Optics Express, 2022, 30(21): 37484-34792). It can be seen that the imaging resolution of the system can be improved by introducing digital micromirror devices, and the combination of digital micromirror devices and array single-photon detectors can improve the acquisition speed while improving the image resolution. However, general data acquisition cards cannot adapt to this complex system control logic, nor can they achieve strict synchronization between data, making it difficult to meet the requirements of real-time and portability. Summary of the invention

[0004] In response to the above problems, the present invention provides a kernel-state and user-state data processing method for an integrated single-photon imaging system, which solves the problem of real-time processing of special data collected by the single-photon lidar system, helps to enhance the integration and reliability of the system and expand its application field.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A kernel state and user state data processing method for an integrated single-photon imaging system, wherein the integrated single-photon imaging system includes a digital micromirror device, an array-type single-photon detector, and a main controller. The light signal is modulated by the small mirror of the digital micromirror device by rapid tilt, and the array-type single-photon detector is used to realize high-resolution image display. The main controller is equipped with a Linux operating system, connected to the digital micromirror device and the array-type single-photon detector, and used to send control and synchronization signals. The high efficiency and open source characteristics of the Linux operating system are used to enable the system to flexibly configure and optimize the data processing process. At the same time, the method includes: collecting a preset mask pattern in the user state, projecting the mask pattern according to preset projection parameters, transmitting the collected data and specific synchronization signals of the array-type single-photon detector to the kernel state for data packaging and compression, selecting a single channel or multiple channels for processing according to the kernel space occupancy rate, and processing the photon data and specific synchronization data during the mask projection in the user state to realize efficient data reconstruction and imaging. The kernel state and user state data processing method includes the following steps:

[0007] Step S1, generating a Hadamard mask according to a preset mask size and based on a Hadamard matrix, and the main controller collects the generated Hadamard mask in a user state; the main controller is equipped with a Linux operating system, and the operating environment is divided into a user state and a kernel state. The user state is a restricted environment for running applications, and the kernel state is a privileged operating mode of the operating system kernel, which has full hardware access rights.

[0008] Step S2, the main controller falls into the kernel state and calls the driver to output the preset related projection parameters to the digital micromirror device. After the digital micromirror device is configured, it plays the Hadamard mask and obtains the photon data and synchronization data when the mask is projected from the array single photon detector; the digital micromirror device is used for image projection, receives the light signal, and uses the micro mirror to quickly tilt and modulate the light signal according to the preset projection parameters to project the mask pattern, so as to achieve high-resolution image display, and at the same time transmits the synchronization data corresponding to the projection mask to the array single photon detector. The projection parameters include: the start frame of the playback sequence, the number of masks to be played, and the playback mode.

[0009] Step S3, in the kernel state of the main controller, the driver program matches the arrayed single-photon detector and receives data, and in the user state, the data is extracted by calling the relevant library function and the extracted photon data and synchronization data during mask projection are processed in combination with the imaging algorithm, and finally a reconstructed image is obtained, wherein the arrayed single-photon detector is used to receive the echo photon signal coupled with the digital micromirror device, and at the same time, the photon data during mask projection is collected, and the photon data and synchronization data during mask projection are packaged and transmitted to the main controller;

[0010] Furthermore, in step S3, the specific steps of the main controller receiving data from the array single photon detector are:

[0011] Step S3.1, matching the corresponding firmware according to the device information of the array single-photon detector, performing security verification on the firmware and storing it in the firmware library, and storing the used firmware information in the firmware library index table to assist in better management of the firmware library and facilitate the next call of the same model device;

[0012] Step S3.2, the main controller establishes a communication connection with the array single photon detector;

[0013] Step S3.3, receiving the photon data and synchronization data during mask projection from the array single-photon detector, and storing them after lossless compression, wherein a timestamp is added to the compressed package according to the timestamp information in the projection synchronization data.

[0014] Step S3.4: During the storage process, the kernel state of the main controller maintains a memory index table to record the location of each compressed package and sort them according to the timestamp information. When the image needs to be reconstructed later, the order and time interval of each compressed package are determined according to the sorting results.

[0015] Furthermore, in step S3, the computing steps of the imaging algorithm specifically include:

[0016] Step S3.5, based on the M frames of photon data collected by the array single-photon detector, perform accumulation operation on each frame to construct an α array with a dimension of M×1; establish a known M×N φ array based on the Hadamard mask; To meet the extended resolution requirement, the pseudo-inverse matrix φ' of the φ matrix is ​​subjected to matrix operation with the α matrix, that is, φ'×α=X, and the required X matrix with dimension N×1 is obtained; according to the dimension N of the obtained X matrix, it is rearranged to obtain This method works in user mode and uses Linux's efficient computing library to support fast matrix operations and improve the execution efficiency of the imaging algorithm. Where M is the number of frames of collected photon data, N is the total number of pixels in the reconstructed image, is the resolution of the reconstructed image, and φ is the known Hadamard mask matrix.

[0017] Step S3.6 monitors the real-time space occupancy of the kernel space in the main controller, and compares the real-time space occupancy with the preset space occupancy threshold. If the real-time space occupancy is ≥ the space occupancy threshold, the backup channel for reconstructing the image is opened. Otherwise, a single set of reconstruction image channels is maintained. Linux's efficient scheduling mechanism makes resource utilization more reasonable and ensures stable system operation.

[0018] Furthermore, the space occupancy rate threshold is 80%.

[0019] The beneficial effects of the present invention are:

[0020] (1) The present invention introduces a Hadamard mask into a digital micromirror device for projection, uses an array-type single-photon detector to collect each projected mask image, and processes multiple frames of images in combination with an imaging algorithm to obtain a reconstructed image, thereby achieving higher imaging resolution, improving imaging quality and the real-time performance of data processing.

[0021] (2) The main controller of the present invention manages memory by carrying a Linux operating system and effectively controls and receives data from digital micromirror devices and array single-photon detectors, thereby ensuring an efficient data processing process, solving the shortcomings of existing single-photon lidar data acquisition systems in terms of special signal processing requirements and enhancing the system's integration. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A flowchart of a kernel state and user state data processing method for an integrated single-photon imaging system proposed by the present invention;

[0023] Figure 2 An imaging schematic diagram of the kernel state and user state data processing method for an integrated single-photon imaging system proposed by the present invention;

[0024] Figure 3 A schematic diagram of the workflow of the kernel state and user state data processing method for an integrated single-photon imaging system proposed by the present invention; DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] The present invention provides a kernel state and user state data processing method for an integrated single-photon imaging system, which solves the problem of real-time processing of special data collected by the single-photon laser radar system, helps to enhance the integration and reliability of the system and expand its application field.

[0027] like Figure 1 As shown, the specific technical solution adopted in the embodiment of the present invention includes the following steps:

[0028] Step S1, generating a Hadamard mask based on a Hadamard matrix according to a preset mask size, and the main controller collects the generated Hadamard mask in a user state. The main controller is equipped with a Linux operating system, and the operating environment is divided into two states: user state and kernel state. The user state is a restricted environment for running applications, and the kernel state is a privileged operating mode of the operating system kernel with full hardware access rights.

[0029] Step S2, the main controller falls into the kernel state and calls the driver program to output the preset related projection parameters to the digital micromirror device. After the digital micromirror device is configured, it plays the Hadamard mask and obtains the photon data and synchronization data when the mask is projected from the array-type single-photon detector; the digital micromirror device is used for image projection, receives the light signal, and uses the tiny mirror to quickly tilt and modulate the light signal according to the preset projection parameters to project the mask pattern for realizing high-resolution image display, and at the same time transmits the synchronization data corresponding to the projection mask to the array-type single-photon detector, and the projection parameters include: the start frame of the playback sequence, the number of playback masks and the playback mode.

[0030] Step S3, in the kernel state of the main controller, the driver program matches the arrayed single-photon detector and receives data, and in the user state, the data is extracted by calling the relevant library function and combined with the photon data and synchronization data obtained during the mask projection by the imaging algorithm to obtain a reconstructed image, wherein the arrayed single-photon detector is used to receive the echo photon signal coupled with the digital micromirror device, and collect the photon data during the mask projection, and the photon data and synchronization data during the mask projection are packaged and transmitted to the main controller;

[0031] In this embodiment, the projection parameters configured for the digital micromirror device according to step S2 specifically include: the start frame of the playback sequence, the number of playback masks and the playback mode;

[0032] Specifically, set the communication interface IP address and process ID, and run the connection detection program. If successful, return 1, otherwise repeat the above process. After the device is successfully connected, send the mask to the flash memory of the digital micromirror device. If successful, return a success signal. If failed, return the error reason and re-execute the above process. After the mask is sent successfully, set the parameters: PlayOffset play start frame = 1; PicCount play number = mask number; PlayModel play mode = 1. After the parameters are set successfully, send a start signal to play the mask. At the same time, output each mask change time point as a synchronization signal node to the array single photon detector, and the subsequent data read from the array single photon detector contains this synchronization signal. When all the masks are played, send an initialization signal to the digital micromirror device, and repeat the play of the group of masks again to achieve continuous acquisition. If you need to stop, send a stop signal to the digital micromirror device to end the mask playback.

[0033] In this embodiment, the specific operation of the main controller receiving data from the array single-photon detector according to step S3 is:

[0034] Step S3.1, matching the corresponding firmware according to the device information of the array single-photon detector, performing security verification on the firmware and storing it in the firmware library, and storing the used firmware information in the firmware library index table to assist in better management of the firmware library and facilitate the next call of the same model device;

[0035] Specifically, according to the specific model of the array-type single-photon detector used, confirm the hardware version and serial number, as well as the type of communication interface used by the device. In this embodiment, the array-type single-photon detector uses a USB3.0 communication interface; according to the hardware model and version number, select a compatible firmware version and install it. After confirming that it has passed the actual use verification, use the hash value verification method to perform security verification, and then store the firmware in the firmware library, and add detailed data information to each firmware file stored in the firmware library. After the firmware is used, the information is synchronously stored in the firmware library index table jointly maintained by the database, so that it can be directly imported from the firmware library when it is called next time, and at the same time, it is ensured that the firmware in the firmware library is available, which helps to better manage and maintain the firmware in the firmware library.

[0036] Step S3.2, the main controller establishes a communication connection with the array single photon detector;

[0037] Step S3.3, receiving the photon data and synchronization data of the mask projection from the array single-photon detector and storing them after lossless compression, wherein a timestamp is added to the compressed package according to the timestamp information in the projection synchronization data.

[0038] Step S3.4: During the storage process in the kernel state of the main controller, a memory index table is maintained to record the location of each compressed package and sort them according to the timestamp information. When the image needs to be reconstructed later, the order and time interval of each compressed package are determined based on the sorting results.

[0039] Specifically, the kernel program converts the received synchronization data into a specific format and embeds it into the header of the data packet to be compressed. The compression program compresses the data with a specific timestamp and stores the compressed data in a pre-set kernel cache. During the storage process, a memory index table is maintained to record the storage location, size and corresponding timestamp information of each compressed packet. When imaging processing is required, the kernel program first queries the memory index table and sorts the compressed packets in ascending order according to the timestamp information. According to the sorting result, the order of calling each compressed packet is determined. The kernel program calls the compressed packets one by one from the kernel cache into the temporary buffer. At the moment when the compressed packets are called into the temporary buffer, the decompression program is automatically triggered.

[0040] It should be noted that the compression program uses the LZ4 algorithm and other efficient lossless compression algorithms.

[0041] In this embodiment, the operation steps of the imaging algorithm according to step S3 specifically include:

[0042] Step S3.5, the data collected by the array single-photon detector is decompressed into M frames of photon data, where the number of frames corresponds to the number of masks, and each frame contains Q photon data. The M frames of data are accumulated for each frame to obtain an α matrix with a dimension of M×1. At the same time, a known M×N φ matrix is ​​established based on the Hadamard mask matrix, where M represents the number of frames collected by the array single-photon detector and N is equivalent to the product of the expanded pixels. Resolution expanded to For example, the X matrix with dimension N×1 is needed, which is equivalent to finding the pseudo-inverse matrix of the φ matrix. The formula φ'×α=X is used for calculation, that is, the pseudo-inverse matrix φ' of the φ matrix is ​​matrix-operated with the α matrix to obtain the X matrix. For example, M=512, The dimension of the φ matrix is ​​512×4096, and in this example the dimension of the X matrix is ​​4096×1. Finally, the 4096 pixels are arranged in sequence to obtain a 64×64 pixel image.

[0043] Where M is the number of frames of collected photon data, Q is the total number of pixels of the original image obtained by the array single-photon detector, and N is the total number of pixels of the final generated image. is the resolution of the original image obtained by the array single-photon detector, is the resolution of the final generated image, and φ is the known Hadamard mask matrix.

[0044] In this embodiment, according to step S3, the main controller receives the array single photon detector data in kernel state and performs imaging algorithm processing in user state. The specific operations include:

[0045] Step S3.6, monitor the real-time space occupancy of the kernel space in the main controller, compare the real-time space occupancy with the preset space occupancy threshold, if the real-time space occupancy ≥ the space occupancy threshold, then open the backup channel for reconstructing the image, otherwise, keep a single set of reconstruction image channels, Linux's efficient scheduling mechanism makes resource utilization more reasonable and ensures stable system operation.

[0046] Specifically, in this embodiment, an independent monitoring thread is run in the kernel state, and the thread checks the occupancy of the kernel space at a fixed time period. The monitoring thread obtains the memory usage of the kernel cache area in the kernel space by querying the memory management interface provided by the operating system, and then calculates the total kernel space occupancy rate. When the monitored kernel cache area occupancy rate reaches a preset threshold of 80%, the monitoring thread sends a signal to the task scheduling module of the kernel space to trigger the dual-channel opening mechanism of the imaging processing. The task scheduling module alternately allocates the photon data when the mask is projected to the two channels according to the timestamp information to realize parallel processing of the data.

[0047] In the single-photon imaging system of this embodiment, the two channels can perform operations such as compressed package retrieval, decompression, and imaging algorithm processing in parallel, and finally merge and optimize the partial reconstructed image results generated by each channel to obtain the final reconstructed image.

[0048] Figure 2 This is an imaging schematic diagram of the kernel state and user state data processing method for the integrated single-photon imaging system in this embodiment. It can be seen from the figure that the main devices included are: digital micromirror device, array single-photon detector, and main controller.

[0049] Specifically, in this embodiment, the main controller uses RK3568 as the core processor and is equipped with a Linux operating system. Its open source characteristics enable it to flexibly adapt to various hardware and software components, enhancing the scalability and compatibility of the system; first, the preset Hadamard mask is packaged and stored in the RK3568 main controller, and the front-end optical path reflects the echo light signal into the digital micromirror device. After the system is started, the RK3568 main controller inputs the mask together with the preset projection parameters into the digital micromirror device. After the digital micromirror device receives the mask, it uses the rapid tilt of the micromirror to control the reflection of light according to the preset projection parameters to achieve high-resolution images. At the same time, the modulated echo photon signal is coupled into the array single-photon detector, and the synchronization data is output to the array single-photon detector according to the time point of each mask projection. The photon data collected during the mask projection together with the corresponding synchronization data are packaged and sent to the RK3568 main controller. The RK3568 main controller uses the imaging algorithm to reconstruct the received photon data and displays the image in real time.

[0050] The self-starting workflow of the kernel state and user state data processing method for an integrated single-photon imaging system according to a specific embodiment of the present invention is as follows: Figure 3As shown in the figure, after each module in the system is powered on, the remaining logs and data files are cleaned up, and the codes of each function are compiled and linked into executable files. At this time, the connection status between the main controller and the array single-photon detector is detected. If the connection is wrong, an error log is generated and the system exits. Under normal conditions, the connection is successful, and the acquisition program is executed first. The RK3568 main controller reads multiple frames of data from the buffer of the array single-photon detector. At the same time, the kernel monitoring thread starts with the system implementation and runs in the background. According to the kernel space occupancy rate, a single channel or dual channel is selected for data processing. The image is reconstructed and displayed on the display screen through the script program. It should be noted that there is a delay setting inside the script. Since it takes time to process and fully display each frame of the collected data each time, the delay setting is to avoid the multi-frame data collected last time not being fully parsed or fully displayed, and the next data will be covered and accumulated again, resulting in errors. At the same time, it also contains an interrupt detection loop to interrupt the loop execution, which is convenient for developers to debug manually.

[0051] Those skilled in the art will appreciate that the above-described embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.

Claims

1. A kernel state and user state data processing method for an integrated single-photon imaging system, wherein the integrated single-photon imaging system includes a digital micromirror device, an array single-photon detector, and a main controller, characterized in that: The kernel state and user state data processing method comprises the following steps: Step S1, generating a Hadamard mask based on a Hadamard matrix according to a preset mask size, the operating system of the main controller divides the environment into two states, a user state and a kernel state, and collects the generated Hadamard mask in the user state; Step S2, the main controller falls into the kernel state and calls the driver to output the preset related projection parameters to the digital micromirror device. After the digital micromirror device is configured, it plays the Hadamard mask and obtains the photon data and synchronization data during the mask projection from the array single photon detector; Step S3, the driver in the kernel state of the main controller matches the arrayed single-photon detector and receives data, and in the user state, the data is extracted by calling the relevant library function and the extracted photon data and synchronization data during mask projection are processed in combination with the imaging algorithm to finally obtain a reconstructed image. The arrayed single-photon detector is used to receive the echo photon signal coupled with the digital micromirror device, and at the same time collect the photon data during mask projection, and the photon data and synchronization data during mask projection are packaged and transmitted to the main controller.

2. The kernel state and user state data processing method for an integrated single photon imaging system according to claim 1, characterized in that: In the step S1, the main controller is equipped with a Linux operating system; the user state is a restricted environment for running application programs, and the kernel state is a privileged operating mode of the operating system kernel with hardware access rights.

3. The kernel state and user state data processing method for an integrated single photon imaging system according to claim 1, characterized in that: In step S2, the digital micromirror device is used for image projection, receives light signals, and projects a mask pattern according to preset projection parameters to achieve high-resolution image display, while transmitting synchronous data corresponding to the projection mask to the array single-photon detector.

4. The kernel state and user state data processing method for an integrated single photon imaging system according to claim 1, characterized in that: In step S2, the projection parameters include: a start frame of a play sequence, a number of play masks, and a play mode.

5. The kernel state and user state data processing method for an integrated single photon imaging system according to claim 1, characterized in that: In step S3, the specific steps of the main controller receiving data from the array single photon detector are: First, the corresponding firmware is matched according to the device information of the array single-photon detector, the firmware is securely verified and stored in the firmware library, and the used firmware information is stored in the firmware library index table; Secondly, the main controller establishes a communication connection with the array single-photon detector; Thirdly, receiving the photon data and synchronization data during mask projection from the array single-photon detector, and storing them after lossless compression, wherein a timestamp is added to the compressed package according to the timestamp information in the projection synchronization data; Finally, during the storage process, the kernel state of the main controller maintains a memory index table to record the location of each compressed package and sort them according to the timestamp information. When the image needs to be reconstructed later, the order and time interval of calling each compressed package are determined based on the sorting results.

6. The kernel state and user state data processing method for an integrated single photon imaging system according to claim 5, characterized in that: In step S3, the computing steps of the imaging algorithm specifically include: First, in the user state, based on the M frames of photon data collected by the array single-photon detector, an accumulation operation is performed on each frame to construct an α array with a dimension of M×1; a known M×N φ array is established based on the Hadamard mask; To meet the extended resolution requirement, the pseudo-inverse matrix φ' of the φ matrix is ​​subjected to matrix operation with the α matrix, that is, φ'×α=X, and the required X matrix with dimension N×1 is obtained; according to the dimension N of the obtained X matrix, it is rearranged to obtain Pixel reconstructed image; where M is the number of frames of collected photon data, N is the total number of pixels in the reconstructed image, is the resolution of the reconstructed image, φ is the known Hadamard mask matrix; Secondly, monitor the real-time space occupancy of the kernel space in the main controller, and compare the real-time space occupancy with the preset space occupancy threshold. If the real-time space occupancy ≥ the space occupancy threshold, open the backup channel for reconstructing the image, otherwise, maintain a single set of reconstructing the image channel.

7. The kernel state and user state data processing method for an integrated single photon imaging system according to claim 6, characterized in that: The space occupancy rate threshold is 80%.

Images