Method and storage medium suitable for high-speed transmission of scientific-grade camera images

Abstract

The present invention provides a method and storage medium suitable for high-speed transmission of scientific-grade camera images. The method comprises the following steps: performing information transmission tasks and data processing tasks separately through asynchronous programming; performing information transmission through a rotation and pre-buffering method; and temporarily storing data on a disk to facilitate block-by-block and position-based access to images. The present invention decouples the transmission process from the data processing process. The transmission process must be processed as quickly as possible to avoid buffer overflows; separating the two through an asynchronous method can prevent the processing process from blocking the transmission process. Pre-buffering avoids executing time-consuming and unstable memory space request operations within the transmission thread during the transmission process, significantly improving the predictability of the system. Information is transmitted in asynchronous tasks using shared memory, avoiding copying and improving efficiency. Token encapsulation provides a clear structure and allows the transmission task and the processing task to exchange important control information.

Description

Technical Field

[0001] The present invention relates to the field of computer technology and equipment control technology, and in particular to a method suitable for high-speed transmission of scientific-grade camera images. Background Art

[0002] The image transmission process of scientific-grade cameras involves a large amount of data input and output. With the expansion of the imaging surface size, the increase in readout channels, the increase in readout speed, and the increase in readout frame rate, the data transmission speed between the camera and the control system is also rapidly increasing. For example, a 4K x 4K imaging detector with 16-bit sampling per pixel can achieve a transmission rate of 2.5 Gbps at 10 fps and 6.4 Gbps at 25 fps. When these data packets are transmitted to the host computer via the fiber optic network card, if the transmission process is not properly designed and matched to the transmission speed, the buffer will fill up, frequent packet loss will occur, and the inability to receive complete image data will be affected, compromising system performance and reliability.

[0003] Regardless of the communication stack used (e.g., the Linux kernel network stack or the libusb-based USB communication stack), high-throughput, high-speed IO processes always involve copying data from the communication stack's private memory space to the transmitting process's memory space. For example, for network transmission using the Linux network kernel stack, this process involves copying data from the Linux kernel buffer to user-mode memory space. However, due to the enormous amount of data, heap memory is always required, making it impossible to store the transmitted data on the stack. This involves the dynamic allocation and release of memory. General-purpose memory allocation and release libraries have long been a hot topic in computer architecture. For example, the general-purpose SLAB algorithm and the tcmalloc library have improved the efficiency of memory allocation and release in general-purpose environments.

[0004] Another issue that must be considered when transmitting images from scientific-grade cameras is that, depending on the sensor, images are typically read out in parallel across multiple readout channels to speed readout. This interweaves the pixel data from each readout channel. However, each readout channel physically corresponds to a different sensor region, which is not actually interwoven. Therefore, after image data is transmitted to the host computer, it undergoes a data reformatting process to form a complete image (a process we define as image stitching). This involves dispatching the interwoven channel data to its corresponding physical locations, arranging them in a row-by-column order, before storing them on disk. This process consumes significant computing power, requires frequent memory accesses, and involves numerous disk read and write operations, making it time-consuming. Paradoxically, image transmission requires software to quickly retrieve data from the kernel buffer to avoid buffer accumulation and exhaustion, which can lead to packet loss. Therefore, transmission tasks must prioritize responsiveness. These two requirements conflict.

[0005] Considering that the data packets in the image transmission process of scientific-grade cameras are usually of fixed size, the total memory consumption can often be predicted based on the image transmission rate. Therefore, a dedicated transmission process framework can be designed based on the characteristics of the image data packet information, and a corresponding memory management mechanism can be used to improve transmission and memory management efficiency. Summary of the Invention

[0006] The present invention proposes a method for high-speed transmission of scientific-grade camera images, which can solve at least one of the above-mentioned technical problems.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] A method for high-speed transmission of scientific-grade camera images includes three aspects:

[0009] Asynchronous programming scheme, information transmission scheme and image caching scheme;

[0010] The significance of the asynchronous programming solution lies in ensuring the execution efficiency of the transmission task and the data processing task by implementing them in different threads, thereby preventing the image processing task from affecting the real-time performance of the transmission task. This asynchronous programming paradigm that uses different threads to support different behaviors requires corresponding technical solutions.

[0011] The significance of the information transmission scheme lies in addressing the problem of large amounts of data (often up to hundreds of megabytes) transferred between different threads and the high overhead of copying this data. By using round-robin and pre-buffering, more efficient memory management is achieved;

[0012] Finally, the image caching solution provides a standard for temporarily storing data on disk, making it convenient to access the image content in blocks and by location; this invention is particularly applicable to situations in scientific-grade camera control software where multi-threaded concurrency is often required and large amounts of data need to be exchanged between threads.

[0013] To support this design, a supporting software package was developed. This package combines the starting address of the memory used to store image data packets, the data packet's metadata, a mutex, and a set of control flags into a single structure called a token. On 64-bit systems, the token is 64 bytes in size, and the overhead of copying the token is negligible. Each token is associated with a pre-allocated heap memory area whose size is an integer multiple of the data packet size. This heap memory area is pre-allocated during software initialization using the malloc call, and paging is disabled using the mlock system call. From this point on, the framework has full control over the management of this heap memory area.

[0014] The present invention divides this memory into several segments according to the size of the data packet, records the first address of each segment, and uses tokens distributed on the system heap memory to correspond to them. At the same time, the mode of applying for and releasing memory is different from the general malloc call and free call for memory application. Two queues are used to manage these tokens. These two queues are respectively called the processing queue and the recycling queue. When the transmission thread calls the system call to read from the buffer, it first takes out the token from the recycling queue and writes the data to the corresponding buffer. Once the buffering is completed, the token is pushed into the processing queue (which is equivalent to releasing this piece of memory); when the processing thread has a token in the processing queue, it takes out the token from it, processes the data in sequence, and finally pushes the token into the recycling queue. During the entire transmission process, only the content of the token is copied, and the memory space that actually stores the image data is never copied, thus saving time.

[0015] To prevent the transmission thread's response speed from being affected by a lack of available tokens in the recycle queue, a pre-buffered token is pushed into the recycle queue based on empirical data before the transmission begins. This approach avoids the need for operating system memory allocation and release calls during the transmission process, thus avoiding the uncertainty in execution time that this process introduces.

[0016] The transmission task calculates the execution status of the transmission process based on the packet header information. Because the header information is very short, the calculation process is usually simple, and the time consumed by these simple calculations has a negligible impact on the response time of the transmission process. The flag information in the token can help the transmission task send special information to the processing task, such as stopping the processing process.

[0017] As can be seen from the above technical solution, the method of the present invention for high-speed transmission of scientific-grade camera images lies in the decoupling of the transmission process and the data processing process. The transmission process must be processed as quickly as possible to avoid buffer overflow; the processing process requires completing the sorting and processing of the data stream, which is relatively time-consuming. Using an asynchronous method to separate the two can prevent the processing process from blocking the transmission process. In addition, the pre-buffering method avoids the execution of time-consuming and unstable memory space request operations within the transmission thread during the transmission process, significantly improving the predictability of the system. The use of shared memory to transmit information in asynchronous tasks avoids copying, thereby improving efficiency. In addition, the use of token encapsulation makes the structure clear and allows the transmission task and the processing task to transmit important control information to each other.

[0018] The method of the present invention, which is suitable for high-speed transmission of scientific-grade camera images, designs a dedicated transmission mechanism based on the traffic characteristics of the image transmission process of scientific-grade cameras and in combination with specific usage scenarios, thereby reducing the impact of the overhead of allocating and releasing memory on the transmission process. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a block diagram of an embodiment of the present invention;

[0020] Figure 2 is a sequence diagram of the exposure process according to an embodiment of the present invention;

[0021] Figure 3 Schematic diagram of data flow during the exposure process of an embodiment of the present invention;

[0022] Figure 4 Schematic diagram of the cache space structure according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0024] like Figure 1 As shown, the method for high-speed transmission of scientific-grade camera images proposed in an embodiment of the present invention includes three aspects:

[0025] Asynchronous programming scheme, information transmission scheme and image caching scheme;

[0026] The significance of the asynchronous programming solution lies in ensuring the execution efficiency of the transmission task and the data processing task by implementing them in different threads, thereby preventing the image processing task from affecting the real-time performance of the transmission task. This asynchronous programming paradigm that uses different threads to support different behaviors requires corresponding technical solutions.

[0027] The significance of the information transmission scheme lies in addressing the problem of large amounts of data (often up to hundreds of megabytes) transferred between different threads and the high overhead of copying this data. By using round-robin and pre-buffering, more efficient memory management is achieved;

[0028] Finally, the image caching solution provides a standard for temporarily storing data on disk, making it convenient to access the image content in blocks and by location; this invention is particularly applicable to situations in scientific-grade camera control software where multi-threaded concurrency is often required and large amounts of data need to be exchanged between threads.

[0029] Specifically, if Figure 2 As shown, the method for high-speed transmission of scientific-grade camera images described in this embodiment includes:

[0030] S1: Initialization: Before starting the transmission task, the initialization process is performed first. The initialization process includes the following parts:

[0031] S1.a dispatches a transmission task thread Tt and a processing task thread Tp; each has its own heap space and can access the same memory address space. Two thread-safe queues are established between the two threads. Both queues are unidirectional. The queue from Tt to Tp is called the processing queue, denoted by Qp; the queue from Tp to Tt is called the recycling queue, denoted by Qr.

[0032] S1.b Pre-buffer: The initialization process first requires allocating a continuous memory space. This memory space is evenly segmented according to the packet size, and the starting address of each segment is recorded. A batch of tokens are created and pushed into Qr, where each token corresponds to the starting address of a memory segment.

[0033] S1.c File Memory Mapping: To avoid consuming additional buffer space when caching the processed target data, and considering that the data will ultimately be cached to disk, file memory mapping is performed during the initialization phase. First, a file with the same size as the data to be cached is created using the fopen call and the fseek call to set the file offset. The file descriptor is then obtained and the file is memory mapped using the mmap system call. The file is then associated with a pointer, called Pdump. Subsequent operations on Pdump are directly mapped to file operations.

[0034] S2. Transmission Process: After completing the above steps, the transmission task begins. In the transmission task, a token, C, is first dequeued from Qr. Before calling the transmission interface, the mutex C.mtx recorded in the token is obtained to avoid potential contention. Subsequently, the address C.addr recorded in the token is passed as a parameter to the transmission interface, causing the kernel transmission layer to copy the data information into the buffer pointed to by the address C.addr recorded in the token.

[0035] S3.Tt releases the mutex C.mtx recorded in the token, and token C is queued into Qp.

[0036] S4. Processing: Tp uses polling to check whether Qp is empty. If Qp contains a token, the first token in Qp is dequeued, denoted as C. First, C.mtx is obtained. The corresponding data packet can then be directly read from the memory space corresponding to the address C.addr recorded in the token. The data stream is then parsed and processed as needed, and the data is written to the space pointed to by Pdump to complete the storage. The Pdump buffer structure is as follows: First, the data packets are separated according to the read channel based on their logical location information. The data for each channel is stored in adjacent spaces on the disk, first in rows and then in columns.

[0037] When the data associated with C is processed, Tp releases the mutex C.mtx recorded in the token, sets each control bit of C to 0, and puts Tp into the queue Qr.

[0038] S5.Tt can use the control bits in the token to control Tp's behavior: When an emergency stop is required, Tp sets the stop flag C.stop of the next token C to 1, instructing Tp to make an emergency stop. When Tt finds that the pre-buffered data space is about to be exhausted, it sets the expansion buffer space control bit C.expand of the next token C to 1, instructing Tp to call the expansion memory space quality, apply for memory space, and generate a new batch of corresponding tokens. In addition, Tt can set the synchronization buffer control bit C.flush of token C to 1 to instruct Tp to synchronize the buffer immediately.

[0039] S6. Clear Operation. When all the above operations are completed and the transmission process ends, Tt pushes a token with C.stop set to 1 into Qp and then exits. After receiving this token, Tp clears all buffers and exits the process.

[0040] A significant advantage of the above-mentioned process of the embodiment of the present invention is the decoupling of the transmission process and the data processing process. The transmission process must be processed as quickly as possible to avoid buffer overflow; while the processing process needs to complete the sorting and processing of the data stream, which is relatively time-consuming; using an asynchronous method to separate the two can avoid the processing process blocking the transmission process. In addition, by pre-buffering, the execution of time-consuming and unstable memory space request operations in the transmission thread during the transmission process is avoided, which significantly improves the predictability of the system. The use of shared memory to transmit information in asynchronous tasks avoids copying, thereby improving efficiency. In addition, the use of token encapsulation makes the structure clear and allows the transmission task and the processing task to transmit important control information to each other.

[0041] The following examples illustrate:

[0042] Taking the application of the present invention in the SDK for the prime camera of the Large Field of View Survey Telescope as an example, a specific implementation scheme of the present invention will be described. The scientific sensor portion of the prime camera of the Large Field of View Survey Telescope consists of nine CCD chips, with a total data volume of 1.5 GB. Allocating memory space commensurate with this total data volume would consume a significant amount of memory resources, hence the use of the present invention. As mentioned above, before the exposure begins, memory space must be pre-prepared and then maintained by the software. Based on practical experience, on a typical host with a 3.5 GHz clock speed, dual cores, four threads, and DDR4 memory, a pre-buffer memory size of 100 MB can meet the scheduling requirements of the entire transmission process.

[0043] After exposure begins, the host computer program sends a request to the camera to read image data. When the camera completes exposure, the image data is transmitted to the host computer. This data is packaged into 8192-byte UDP packets, with metadata added to the header. The host computer uses the UNIX system read call to copy data from the socket to user memory, using the memory space pointed to by the top token in the recycling queue described above. After calculating the current state based on the metadata in the packet header, the token containing the data frame is pushed to the processing queue.

[0044] The processing thread uses conditional variables to realize the feature of being awakened when the processing queue is not empty; when the transmission thread pushes the token containing the data frame to the processing queue, the processing thread wakes up and starts to split and summarize the data by channel and write it into the memory space obtained by mmap file memory mapping. The specific organization method is as follows Figure 3 As shown; once the writing is completed, the corresponding token will be pushed into the recycling queue for use by the transmission thread.

[0045] The transmission thread uses the packet header metadata to calculate the transmission status and, based on the packet count, determines whether the image data has been completely transmitted to the host computer. If not, it starts the retransmission process. If all transmissions are completed, a token with the stop flag set to True is pushed to the processing queue. When the processing process encounters this token, it stops waiting for data to arrive, cleans up resources, and ends the transmission process. In this way, a complete exposure process is completed.

[0046] In another aspect, the present invention further discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of any of the above methods.

[0047] On the other hand, the present invention further discloses a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any of the above methods.

[0048] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the steps of any one of the methods in the above embodiments.

[0049] It is understandable that the system provided by the embodiment of the present invention corresponds to the method provided by the embodiment of the present invention, and the explanation, examples and beneficial effects of the relevant contents can refer to the corresponding parts of the above method.

[0050] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0051] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0052] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for high-speed transmission of scientific-grade camera images, characterized in that: The following steps are involved: Through asynchronous programming, information transmission tasks and data processing tasks are carried out separately; Information is transmitted through round-robin and pre-buffering methods; Temporarily store data on disk to facilitate block-by-block and position-based access to images; The first address of the memory used to store the image data packet, the metadata of the data packet, a mutex, and a set of control flags are unified to form a structure called a token; The following steps are involved: S1: The initialization process includes the following steps: S1a, dispatch the transmission task thread Tt and the processing task thread Tp; each has its own heap space and can access the same memory address space; establish two thread-safe queues between the two threads, both of which are unidirectional. The queue from Tt to Tp is called the processing queue, denoted by Qp; the queue from Tp to Tt is called the recycling queue, denoted by Qr; S1b, Pre-buffer: In the initialization process, a whole continuous memory space is first allocated. This memory space is evenly segmented according to the size of the data packet. The first address of each segment is recorded, and a batch of tokens are created and pushed into Qr. Each token corresponds to the first address of a memory segment. S1c, file memory mapping: perform file memory mapping operations in the initialization phase: first, use fopen to open the file and fseek to set the file offset to create a file with the same size as the data to be cached, and obtain its file descriptor. Then, use the mmap system call to associate the file with a pointer, which is called Pdump. Subsequent operations on Pdump will be directly mapped to operations on the file. S2. Transmission process: After completing the above steps, the transmission task begins. In the transmission task, a token, C, is first dequeued from Qr. Before calling the transmission interface, the mutex C.mtx recorded in the token is obtained. Then, the address C.addr recorded in the token is passed as a parameter to the transmission interface, causing the kernel transmission layer to copy the data information into the buffer pointed to by the address C.addr recorded in the token. S3. Tt releases the mutex C.mtx recorded in the token, and token C is added to queue Qp. S4. Processing: Tp uses polling to check whether Qp is empty. If Qp contains a token, it dequeues the first token in Qp, denoted as C. First, it obtains the mutex C.mtx recorded in the token. Then, it directly reads the memory space corresponding to the address C.addr recorded in the token to obtain the corresponding data packet. The data stream is then parsed and processed as needed, and the data is written to the space pointed to by Pdump to complete the storage. When the data associated with C is processed, Tp releases the mutex C.mtx recorded in the token, sets each control bit of C to 0, and puts Tp into the queue Qr.

2. The method for high-speed transmission of scientific-grade camera images according to claim 1, characterized in that: Each token is associated with a pre-allocated piece of heap memory whose size is an integer multiple of the packet size; These heap memories are allocated in advance by calling malloc when the software is initialized, and page swapping is prohibited for this part of memory by calling mlock system call. This memory is divided into several segments according to the size of the data packet, and its first address is recorded respectively. Tokens distributed on the system heap memory are used to correspond to them. Two queues are used to manage these tokens. These two queues are called processing queue and recycling queue respectively. When the transmission thread calls the system call to read from the buffer, it first takes the token from the recycling queue and writes the data into the corresponding buffer. Once the buffering is completed, the token is pushed into the processing queue; When there is a token in the processing queue, the processing thread takes the token out of it, processes the data in sequence, and finally pushes the token into the recycling queue; During the entire transmission process, only the content of the token is copied, while the memory space where the image data is actually stored is never copied; The transmission task calculates the execution status of the transmission process based on the header information of the data packet.

3. The method for high-speed transmission of scientific-grade camera images according to claim 2, characterized in that: It also includes pre-buffering based on experience before starting image exposure, and pushing some tokens into the recycling queue first.

4. The method for high-speed transmission of scientific-grade camera images according to claim 3, characterized in that: The flag information in the token helps the transmission task to send some set special information to the processing task.

5. The method for high-speed transmission of scientific-grade camera images according to claim 3, characterized in that: The following steps are also included: S5. Tt uses the control bits in the token to control Tp's behavior: When an emergency stop is required, Tp sets the stop flag bit C.stop of the next token C to 1, instructing Tp to make an emergency stop. When Tt detects that the pre-buffered data space is about to be exhausted, it sets the expansion buffer space control bit C.expand of the next token C to 1, instructing Tp to call the expansion memory space function, apply for memory space, and generate a new batch of corresponding tokens. S6. Clear operation: When all the above operations are completed and the transmission process is over, Tt pushes a token with C.stop set to 1 into Qp and then actively exits. After receiving the token, Tp clears all buffers and exits the process.

6. The method for high-speed transmission of scientific-grade camera images according to claim 5, characterized in that: The structural constraints of the Pdump buffer in step S4 are as follows: first, the data packets are separated according to the readout channels based on the logical location information of the data packets. The data of each channel is in adjacent space on the disk and stored in order from row to column.

7. The method for high-speed transmission of scientific-grade camera images according to claim 5, characterized in that: In step S5, Tt can request Tp to synchronize the buffer immediately by setting the synchronization buffer control bit C.flush of token C to 1.

8. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 7.

Images