Wearable microscope system with double processing chips based on WIFI image transmission and transmission method

Abstract

The invention discloses a dual-processing chip wearable microscope system based on WIFI image transmission and a transmission method. The system comprises an image sensor which is used for sensing an optical signal and generating original image data; the lower image acquisition chip is used for reading the original image data, caching the original image data, packaging the original image data into a local image data packet, and forwarding the local image data packet to the upper image acquisition chip; and the upper image acquisition chip is used for checking the data integrity of the local image data packet, packaging the local image data packet into an image frame data packet, and sending the image frame data packet to the control terminal by adopting a WIFI image transmission technology. According to the invention, on the premise that the weight of the wearable microscope system is not obviously increased, the image data processing performance of the wearable microscope system is improved, WIFI image transmission is carried out by taking an image frame as a unit, the wifi bandwidth is matched, the protocol overhead is reduced, the transmission frequency and delay are reduced, and accidental frame loss is avoided.

Description

Technical Field

[0001] This invention belongs to the field of biomedicine, and more specifically, relates to a wearable microscope system and transmission method based on WIFI image transmission using dual processing chips. Background Technology

[0002] Wearable microscopy systems are a miniaturized imaging technology that optically records the activity of neurons in the animal brain, such as calcium signaling activity in nerve cells of freely moving animals. It is an important research method in neuroscience.

[0003] Wearable microscope systems generally include a control module, a data acquisition module, and a data transmission module; data transmission is divided into two main categories: wireless transmission and wired transmission. Wireless wearable microscope systems do not hinder the natural behavior of animals due to the presence or even tangling of cables, and have unparalleled advantages over wired systems, representing the future trend.

[0004] Currently, wireless wearable microscope systems primarily employ three data transmission methods: data storage, analog signal (NTSC mode) transmission, and digital signal transmission. Data storage cannot provide real-time microscopic image acquisition for interactive control. Analog signal transmission has limited spatial resolution and image quality, suffers from rapid signal attenuation, and often limits the working distance to within 3 meters. Digital signal transmission offers high image quality, utilizes digital transmission protocols such as Bluetooth, Wi-Fi, and ZigBee, has a long working distance, and strong anti-interference capabilities, but requires a large amount of data transmission. While Wi-Fi technology offers significant bandwidth to meet the demands of high-resolution image transmission, wireless wearable microscope systems cannot fully utilize its bandwidth advantage, leading to data errors and unexpected frame drops or skips. Directly employing high-performance data processing chips to fully leverage Wi-Fi's bandwidth significantly increases the overall quality of the wearable microscope system, hindering the free movement of the observed animal and causing biased observation results. Summary of the Invention

[0005] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a wearable microscope system based on WIFI image transmission using dual processing chips. The purpose is to package the raw image frame data generated by the image sensor into image frame data packets, with local image data packets as the primary unit and image frames as the secondary unit, and then transmit these packets through a buffering and forwarding mechanism for lossless high-resolution image transmission. This improves the data transmission performance of the wearable microscope system with almost no increase in weight, thereby solving the technical problem that existing wireless wearable microscope systems cannot fully utilize the bandwidth advantage of WIFI technology, making it difficult to meet the high temporal and image resolution observation requirements, including those related to nerve cell activity.

[0006] To achieve the above objectives, according to one aspect of the present invention, a wearable microscope system based on WIFI image transmission with dual processing chips is provided, including an image sensor, a lower-level image acquisition chip, and an upper-level image acquisition chip. The image sensor is used to sense light signals and generate raw image data; the raw image data is a digitally represented pixel array. The lower-level image acquisition chip is used to read the original image data, cache the original image data, package it into a local image data packet, and forward it to the upper-level image acquisition chip; the local image data packet includes a pixel array of a preset area of ​​the original image data, a frame number, and a packet number; the packet number is used to characterize the preset area of ​​the original image where the data packet is located, and has a start packet number and an end packet number according to the area order; the frame number is used to mark the original image where the data packet is located. The upper-level image acquisition chip is used to verify the data integrity of local image data packets and package the local image data packets into image frame data packets, and send the image frame data packets to the control terminal using WIFI image transmission technology. The control terminal wirelessly receives image frame data packets, determines the image timing based on the frame number of the image frame data packets to avoid accidental frame loss, and verifies the data integrity of the local image data packets based on the packet numbers of all local image data packets in the image frame data packets.

[0007] Preferably, in the wearable microscope system based on WIFI image transmission with dual processing chips, the lower-level image acquisition chip uses a cache queue to cache local image data of a preset area of ​​the original image and packages it into a local image data packet. Preferably, the lower-level image acquisition chip provides multiple cache queues to acquire the original image.

[0008] Preferably, in the wearable microscope system based on WIFI image transmission with dual processing chips, the lower-level image acquisition chip replaces edge pixels with frame numbers and / or packet numbers.

[0009] Preferably, in the wearable microscope system with dual processing chips based on WIFI image transmission, the lower image acquisition chip and the image sensor transmit data directly; the lower image acquisition chip and the upper image acquisition chip preferably communicate using the SPI protocol to receive imaging control signals.

[0010] Preferably, in the wearable microscope system based on WIFI image transmission with dual processing chips, the image frame data packet includes all local image data packets of a frame of original image; the verification of the integrity of the local image data packet data specifically involves: determining, based on the frame number and packet number, that the image frame data consists of all local image data packets of a frame of original image.

[0011] Preferably, the wearable microscope system based on WIFI image transmission with dual processing chips uses the lower-level image acquisition chip as the main transmission chip and employs a handshake mechanism to establish a channel according to the control clock generated by the upper-level image acquisition chip for local image data packet buffering and forwarding in units of image frames, as follows: After the lower-level image acquisition chip completes the buffering and packaging of a new local image data packet, it sends a handshake signal to the upper-level image acquisition chip. The upper-level image acquisition chip establishes a data communication channel and generates a periodic transmission clock based on the handshake signal; the transmission duration of the transmission clock is less than the buffer duration of the local image data packets generated by the lower-level machine. The lower-level image acquisition chip sends all local image data packets of the frame image according to the transmission clock generated by the upper-level image acquisition chip; After receiving a preset number of local image data packets, the host image acquisition chip performs packet number verification to ensure the integrity of the local image data packets of the frame image, and packages all the local image data packets of the frame image into an image frame data packet, and sends the image frame data packet to the control terminal using WIFI image transmission technology.

[0012] According to another aspect of the present invention, a method for transmitting image data with dual verification is provided, comprising the following steps: (1) The lower-level image acquisition chip divides each frame of the original image data into local images and packages them into local image data packets, which are then sent to the upper-level image acquisition chip. The local image data packets include a pixel array representing a preset area of ​​the original image data, a frame number, and a packet number. The packet number is used to characterize the preset area of ​​the original image where the data packet is located, and has a start packet number and an end packet number in the order of the areas. The frame number is used to mark the original image where the data packet is located. (2) After receiving the local image data packet, the upper image acquisition chip parses the frame number and packet number of the data packet. If the packet number of the local image data packet is the same as the current expected packet number, the data is cached. Otherwise, the local image data packet is discarded, the cache is cleared, and the expected packet number is reset to the starting packet number. When the packet number of the cached local image data packet is the ending packet number, the cached local image data packet is packaged into an image frame data packet and forwarded to the control terminal. (3) After receiving the image frame data packet, the control terminal parses the frame number of the data packet and compares it with the frame number of the received image frame data packet to check whether there is a missing frame number or a duplicate frame number. If there is a missing frame number, it is judged as a lost frame and the missing frame number is recorded; if there is a duplicate frame number, it is judged as a data retransmission and the image frame data packet is discarded.

[0013] Preferably, in the image data dual-verification transmission method, step (1) involves the lower-level image acquisition chip establishing a channel based on the control clock generated by the upper-level image acquisition chip using a handshake mechanism to perform local image data packet buffering and forwarding in units of image frames; specifically as follows: After the lower-level image acquisition chip completes the buffering and packaging of a new local image data packet, it sends a handshake signal to the upper-level image acquisition chip. The upper-level image acquisition chip establishes a data communication channel and generates a periodic transmission clock based on the handshake signal; the transmission duration of the transmission clock is less than the buffer duration of the local image data packets generated by the lower-level machine. The lower-level image acquisition chip sends all local image data packets of the frame image according to the transmission clock generated by the upper-level image acquisition chip.

[0014] Preferably, in the image data dual verification transmission method, in step (2), the upper-level image acquisition chip determines that all local image data packets in the cached data have the same frame number, then packages the cached local image data packets into image frame data packets; otherwise, it clears the cache and terminates the expected packet number with the starting packet number.

[0015] Preferably, in the image data dual verification transmission method, step (3) involves the upper-level image acquisition chip performing single-frame integrity verification, determining that all local image data packets in the cached data have the same frame number, then packaging the cached local image data packets into image frame data packets; otherwise, clearing the cache and terminating the expected packet number with the starting packet number.

[0016] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects: The wearable microscope system based on WIFI image transmission provided by this invention uses a two-level processing chip for multi-level data packaging and buffering forwarding. It packages the raw image frame data generated by the image sensor into image frame data packets with local image data packets as the primary unit and image frame data packets as the secondary unit. This reduces the storage chip required for WIFI image transmission in image frame bit units, thereby improving the image data processing performance of the wearable microscope system without significantly increasing its weight. WIFI image transmission in image frame units matches the WIFI bandwidth, reduces protocol overhead, lowers the number of transmissions and latency, and avoids accidental frame loss.

[0017] The image data dual-verification transmission method provided by this invention employs a multi-level caching mechanism combined with packet number and check bit dual-verification. At the upper-level image acquisition chip, only packet number verification is performed to ensure data integrity in wireless data transmission at the image frame level. At the high-performance control terminal, frame number verification is used for timing verification of image frame data packets, and packet number verification is used for data integrity verification of local image data packets. This dual-verification avoids data anomalies caused by transmission misalignment and allows for easy tracing of lost data. Abnormal data is controlled within a single image frame, preventing large-scale and prolonged data anomalies. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the transmission of a dual-processing chip wearable microscope system based on WIFI image transmission provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the field of view of a wearable microscope system with dual processing chips based on WIFI image transmission provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the data acquisition and verification process of a dual-processing chip wearable microscope system based on WIFI image transmission provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the signal connection and transmission between the lower-level image acquisition chip and the upper-level image acquisition chip of the dual-processing chip wearable microscope system based on WIFI image transmission provided in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0020] The wearable microscope system based on WIFI image transmission with dual processing chips provided by the present invention includes an image sensor, a lower-level image acquisition chip, and an upper-level image acquisition chip. The image sensor is used to sense light signals and generate raw image data; the raw image data is a digitally represented pixel array. The lower-level image acquisition chip is used to read the original image data, cache the original image data, and package it into a local image data packet, which is then forwarded to the upper-level image acquisition chip. Specifically, the lower-level image acquisition chip uses a cache queue to cache local image data of a preset area of ​​the original image and packages it into a local image data packet. The local image data packet includes a pixel array representing the preset area of ​​the original image data, a frame number, and a packet number. The packet number is used to characterize the preset area of ​​the original image where the data packet is located, and has a start packet number and an end packet number according to the region order. The frame number is used to mark the original image where the data packet is located.

[0021] In a preferred embodiment, the lower-level image acquisition chip replaces edge pixels with frame numbers and / or packet numbers. Since the image data of the imaging system is a square matrix, while the field of view of the microscopic system is generally circular, edge pixels are usually blank information. Replacing edge pixels with packet numbers and frame numbers will not change the length of the local image data packet, which facilitates the data transmission and verification of the system.

[0022] The lower-level image acquisition chip directly transmits data to the image sensor and preferably communicates with the upper-level image acquisition chip via the SPI protocol to receive imaging control signals, such as CMOS control signals. Preferably, the lower-level image acquisition chip provides multiple buffer queues for acquiring raw images. The lower-level image acquisition chip is preferably an FPGA chip. FPGA chips offer flexible data processing capabilities; they can divide the raw image data into pre-defined local image data packets according to the required size and can perform high-frequency data reading and transmission as needed. Furthermore, FPGA chips can freely manipulate the data of each pixel, enabling in-situ replacement of packet numbers and / or frame numbers without changing the length of the data packets. The upper-level image acquisition chip is used to verify the data integrity of local image data packets and package the local image data packets into image frame data packets, and send the image frame data packets to the control terminal using WIFI image transmission technology; the image frame data packet includes all local image data packets of a frame of original image; the verification of the data integrity of local image data packets specifically involves: determining, based on the frame number and packet number, that the image frame data consists of all local image data packets of a frame of original image.

[0023] This invention employs a dual-processing chip architecture to implement a multi-level caching mechanism. It packages the raw image frame data generated by the image sensor into a primary unit of local image data packets and a secondary unit of image frame data packets. Through a caching and forwarding mechanism, it meets the needs of high-frequency image data transmission with minimal storage overhead. Compared to the architecture of a single processing chip paired with a large cache chip, the overall mass of the wearable microscope system is reduced, minimizing interference with experimental animals.

[0024] Meanwhile, the two-level cache packaging data processing mechanism has a smaller primary unit of data volume, which facilitates the location and processing of abnormal data. It avoids the problem of long-term data unavailability caused by data loss and misalignment after direct transmission of image data. That is, after the accidental loss of data packets, the image transmission is completely misaligned, and the data is unavailable for the entire period of time. The transmission channel can only be re-established.

[0025] Because it's necessary to observe subtle neuronal activity in animals, mainstream compression formats are unsuitable for data acquisition and transmission, leading to the loss of useful image information. If raw data is directly packaged into image frames for Wi-Fi transmission, multiple buffer queues are typically used to buffer the data to meet transmission frequency requirements. Each buffer queue needs to match a single frame of raw data, requiring a larger lower-level buffer, significantly increasing device weight and exacerbating interference with experimental animals. While packaging raw data into partial image data packets for wireless transmission can alleviate the frequency difference before and after packaging, it doesn't fully utilize Wi-Fi bandwidth, resulting in excessive protocol overhead and failing to meet the image transmission needs of high-frequency, high-volume wearable microscope systems. Therefore, a dual-processor chip approach is adopted. By adding one more processing chip, the required buffer size for the lower-level image acquisition chip is significantly reduced, while the upper-level image acquisition chip only needs to match the size of a single frame of raw image data, thus reducing the overall weight of the wearable microscope system.

[0026] Furthermore, using the upper-level image acquisition chip as a slave device for communication between the upper and lower-level image acquisition chips leads to uncontrollable signal integrity and timing jitter issues, failing to meet communication speed requirements. The lower-level image acquisition chip generates the data to be transmitted, and generally, it should generate a control clock to control data transmission between itself and the upper-level image acquisition chip. However, the buffer size of the lower-level image acquisition chip is far smaller than that of the control terminal image acquisition chip. If the lower-level image acquisition chip generates the control clock as the master transmission chip, the upper-level image acquisition chip needs to frequently calibrate according to the control clock generated by the lower-level chip. This necessitates real-time sampling of external high-speed clock signals, causing problems with signal data integrity, jitter, and timing constraints between the upper and lower-level image acquisition chips. This is especially problematic at high frequencies, where it becomes almost uncontrollable, frequently resulting in localized image data packet loss. After packet loss, the entire image frame data is discarded, leading to frame loss. To improve the stability of data transmission between the upper and lower image acquisition chips, this invention preferably uses the lower image acquisition chip as the main transmission chip. A handshake mechanism is used to establish a channel based on the control clock generated by the upper image acquisition chip to perform local image data packet buffering and forwarding in units of image frames, as follows: After the lower-level image acquisition chip completes the buffering and packaging of a new local image data packet, it sends a handshake signal to the upper-level image acquisition chip, preferably a Trigger handshake signal, which supports fully asynchronous inter-clock domain data transmission. The upper-level image acquisition chip establishes a data communication channel and generates a periodic transmission clock based on the handshake signal; the transmission duration of the transmission clock is less than the buffer duration of the local image data packets generated by the lower-level machine. The lower-level image acquisition chip sends all local image data packets of the frame image according to the transmission clock generated by the upper-level image acquisition chip; After receiving a preset number of local image data packets, the host image acquisition chip performs packet number verification to ensure the integrity of the local image data packets of the frame image, and packages all the local image data packets of the frame image into an image frame data packet, and sends the image frame data packet to the control terminal using WIFI image transmission technology.

[0027] In a preferred embodiment, the upper-level image acquisition chip, preferably an ESP32 chip, receives imaging control signals and forwards them to the lower-level image acquisition chip and / or performs imaging control.

[0028] The control terminal wirelessly receives image frame data packets, determines the image timing based on the frame number of the image frame data packets to avoid accidental frame loss, and verifies the data integrity of the local image data packets based on the frame numbers of all local image data packets in the image frame data packet.

[0029] The wearable microscope system with dual processing chips provided by this invention offers a method for transmitting image data with dual verification, comprising the following steps: (1) The lower-level image acquisition chip divides each frame of the original image data into local images and packages them into local image data packets, which are then sent to the upper-level image acquisition chip. The local image data packets include a pixel array representing a preset area of ​​the original image data, a frame number, and a packet number. The packet number is used to characterize the preset area of ​​the original image where the data packet is located, and has a start packet number and an end packet number in the order of the areas. The frame number is used to mark the original image where the data packet is located.

[0030] In a preferred embodiment, the lower-level image acquisition chip uses a handshake mechanism to establish a channel based on the control clock generated by the upper-level image acquisition chip, and performs local image data packet buffering and forwarding in units of image frames; specifically as follows: After the lower-level image acquisition chip completes the buffering and packaging of a new local image data packet, it sends a handshake signal to the upper-level image acquisition chip, preferably a Trigger handshake signal, which supports fully asynchronous inter-clock domain data transmission. The upper-level image acquisition chip establishes a data communication channel and generates a periodic transmission clock based on the handshake signal; the transmission duration of the transmission clock is less than the buffer duration of the local image data packets generated by the lower-level machine. The lower-level image acquisition chip sends all local image data packets of the frame image according to the transmission clock generated by the upper-level image acquisition chip; (2) After receiving the local image data packet, the upper image acquisition chip parses the frame number and packet number of the data packet. If the packet number of the local image data packet is the same as the current expected packet number, the data is cached. Otherwise, the local image data packet is discarded, the cache is cleared, and the expected packet number is reset to the starting packet number. When the packet number of the cached local image data packet is the ending packet number, the cached local image data packet is packaged into an image frame data packet and forwarded to the control terminal.

[0031] In a preferred embodiment, if the upper-level image acquisition chip determines that all local image data packets in the cached data have the same frame number, it will package the cached local image data packets into image frame data packets; otherwise, it will clear the cache and terminate the expected packet number with the starting packet number.

[0032] (3) After receiving the image frame data packet, the control terminal parses the frame number of the data packet and compares it with the frame number of the received image frame data packet to check whether there is a missing frame number or a duplicate frame number. If there is a missing frame number, it is judged as a lost frame and the missing frame number is recorded; if there is a duplicate frame number, it is judged as a data retransmission and the image frame data packet is discarded.

[0033] In a preferred embodiment, the control terminal verifies whether all local image data packets in the image frame data packet come from the same true image data. If all local image data packets have the same frame number, the frame number is used as the frame number of the image frame data packet; otherwise, the image frame data packet is discarded.

[0034] This invention employs a multi-level caching mechanism combined with a dual verification method using packet number and check bits. The raw image data generated by the high-frequency image sensor in the lower-level image acquisition chip is marked with frame number and packet number. In the upper-level image acquisition chip, only packet number verification is performed to ensure data integrity in wireless data transmission at the image frame level. At the high-performance control terminal, frame number verification is used for timing verification of image frame data packets, and packet number verification is used for data integrity verification of local image data packets. This dual data verification avoids data anomalies caused by transmission misalignment and allows for easy tracing of lost data, controlling abnormal data within a single image frame to prevent large-scale, long-term data anomalies. The wearable microscope system provided by this invention has minimal computational overhead in the wearable component. The entire system achieves highly controllable integrity verification. By minimizing the computational overhead of step-by-step data integrity verification in the wearable microscope system through multi-level caching and forwarding mechanisms, the performance requirements of the upper-level image acquisition chip are reduced. This is crucial for reducing system thermal noise and meeting the low-interference observation requirements of the wearable microscope system for laboratory animals.

[0035] Since the high-frequency, large-data-volume microscopic image transmission between the host image acquisition chip and the terminal is achieved through WIFI image transmission, automatic data retransmission is performed according to the TCP / IP protocol, which may lead to abnormal image timing and image timing disorder. This invention uses frame number verification to solve the image timing problem. After receiving the image frame data packet, the control terminal obtains the frame number and organizes the image frame data according to the frame number order to avoid data misalignment caused by accidental frame loss.

[0036] The following is an example: The Wi-Fi Image Transmission-based Dual-Processing Chip Wearable Microscope System WIFIMiniscope provided in this embodiment of the invention, such as... Figure 1 As shown, this is a wireless miniature microscope system that can be worn on the head of a freely moving living organism for extended periods to record high-resolution real-time calcium activity of its brain neurons and transmit images via WIFI. The main body of the device weighs relatively lightly at 7.53g, and there are two battery options: a 250mAh battery weighing 6.34g and a 150mAh battery weighing 4.3g. The microscope dimensions are: 33.4mm long, 24.2mm wide, and 30.7mm high. This includes image sensors, lower-level image acquisition chips, and upper-level image acquisition chips; The image sensor is used to sense light signals and generate raw image data; the raw image data is a digitally represented pixel array; the image sensor consists of a CMOS sensor and a lens module.

[0037] The lower-level image acquisition chip is used to read the original image data, cache the original image data, and package it into a local image data packet, which is then forwarded to the upper-level image acquisition chip. Specifically, the lower-level image acquisition chip uses a cache queue to cache local image data of a preset area of ​​the original image and packages it into a local image data packet. The local image data packet includes a pixel array representing the preset area of ​​the original image data, a frame number, and a packet number. The packet number is used to characterize the preset area of ​​the original image where the data packet is located, and has a start packet number and an end packet number according to the region order. The frame number is used to mark the original image where the data packet is located.

[0038] The lower-level image acquisition chip replaces edge pixels with frame numbers and packet numbers. Since the image data of the imaging system is a square matrix, while the field of view of the microscopic system is generally circular, edge pixels are usually blank information. Replacing edge pixels with packet numbers and frame numbers will not change the length of the local image data packet, which facilitates the data transmission and verification of the system.

[0039] Based on the WIFIMiniscope design, since the CMOS image is square and the field of view is circular, as shown below. Figure 2 As shown, pixels at the image edges do not contain useful information. This characteristic can be used to replace edge pixel data with flag bits and check bits, thus adding the necessary information without changing the data frame length, effectively saving system bandwidth.

[0040] The image acquisition process in FPGA is as follows Figure 3 As shown: When data passes through the data acquisition module, if it happens to be the first pixel of the first and second rows of each data packet, the data of that pixel will be discarded and replaced with the packet number and frame number for data verification.

[0041] The lower-level image acquisition chip directly transmits data to the image sensor and preferably communicates with the upper-level image acquisition chip via the SPI protocol, receiving imaging control signals, such as CMOS control signals. The lower-level image acquisition chip provides multiple buffer queues for acquiring raw images. The lower-level image acquisition chip is an FPGA chip. The FPGA acquisition system uses an Altera EP4CE10F17C8 FPGA as the main control chip and is developed using Verilog. Its internal design is mainly divided into four modules: an image acquisition module, a data buffer module, a data transmission module, and a CMOS configuration module. Upon power-up, the FPGA acquisition system configures the CMOS registers via the SPI protocol. Afterward, the image is acquired by the FPGA chip.

[0042] The raw image acquired by the CMOS is an 8-bit grayscale image of 608*608 pixels. The CMOS transmits 8 bits of data (one pixel) to the FPGA at a time until a frame is completed. Each frame is divided into 32 local image data packets in the FPGA, each packet has 19 rows, and each row has 608 pixels. The first two pixels of the first row of the first packet of each frame are marked with the packet number, the first pixel of the second row is marked with the frame number, and the first pixels of the first two rows of the remaining packets are marked with only the packet number and frame number. The pixel data is stored one by one in a buffer with a size of 2 packets, using a ping-pong buffering method. When a buffer is full, the data is forwarded to the ESP32 via the SPI protocol. At the same time, control commands returned by the ESP32 to the user are also received, such as adjusting the CMOS gain, switching the LED on and off, and performing image synchronization operations.

[0043] The upper-level image acquisition chip is used to verify the data integrity of local image data packets and package the local image data packets into image frame data packets, and send the image frame data packets to the control terminal using WIFI image transmission technology; the image frame data packet includes all local image data packets of a frame of original image; the verification of the data integrity of local image data packets specifically involves: determining, based on the frame number and packet number, that the image frame data consists of all local image data packets of a frame of original image.

[0044] The local image data packet buffering and forwarding process is as follows: After the lower-level image acquisition chip completes the buffering and packaging of a new local image data packet, it sends a Trigger handshake signal to the upper-level image acquisition chip. The upper-level image acquisition chip establishes a data communication channel and generates a periodic transmission clock based on the handshake signal; the transmission duration of the transmission clock is less than the buffer duration of the local image data packets generated by the lower-level machine. The lower-level image acquisition chip sends all local image data packets of the frame image according to the transmission clock generated by the upper-level image acquisition chip; After receiving a preset number of local image data packets, the host image acquisition chip performs packet number verification to ensure the integrity of the local image data packets of the frame image, and packages all the local image data packets of the frame image into an image frame data packet, and sends the image frame data packet to the control terminal using WIFI image transmission technology.

[0045] The upper-level image acquisition chip is an ESP32 chip, which receives imaging control signals and forwards them to the lower-level image acquisition chip and / or performs imaging control.

[0046] The FPGA acquisition system and the ESP32 transmission system use the SPI protocol for data transmission, such as... Figure 4 As shown. SPI is a high-speed serial protocol interface. The SPI master needs to provide a data transfer clock (SCLK), chip select signal (CS), MOSI (master output, slave input), and needs to receive data on the MISO signal line; while the slave needs to place the data to be transmitted on the MISO signal line according to SCLK and CS. The ESP32 is configured as the master, and a trigger signal transmitted from the FPGA to the ESP32 is added. This signal will go high after a data packet is accumulated in the FPGA's FIFO. When the ESP32 receives this high level, it will perform an SPI data transfer, transmitting the data packet to the ESP32. Because the data transfer rules of the driver layer have been changed, the transmission protocol is not the traditional SPI protocol, but a new protocol we created to address the limitations of the chip. The signal connection and transmission diagram are shown below. Figure 4 As shown.

[0047] Data received by the ESP32 via SPI is stored in the SPI_Recv_Buffer. After each data packet is received, a verification procedure is initiated. The verification process is as follows: Figure 3As shown. First, the packet number of this data packet needs to be extracted and compared with the expected packet number stored in the packet counter (packet_cnt). If the two values ​​are the same, the verification passes, and this data packet is written from SPI_Recv_Buffer to WIFI_Send_Buffer via the memcpy function. At the same time, packet_cnt increments by 1, reaching a maximum of 32, after which it returns to 1. If the two values ​​are different, the verification fails, packet_cnt is reset to 1, and the previous data in WIFI_Send_Buffer is cleared, waiting for the data with packet number 1 in the next frame. If this verification step is not performed, the packet loss error will accumulate, resulting in the screen splitting vertically. The cause of this packet loss may be due to channel overlap. The device may pause data transmission; however, if data transmission is paused, there is a certain probability that the WIFITask thread will be blocked, causing the ESP32 to fail to recognize the handshake signal sent by the FPGA. The FPGA uses a ping-pong buffer (a ping-pong buffer consists of two FIFOs, each containing one data packet. When one FIFO is writing data, the other is reading data. FIFO1 stores odd-numbered data packets with numbers 1, 3, 5...31, while FIFO2 stores even-numbered data packets). If a trigger signal is not read (for example, if packet number 3 is not read), the buffer read pointer remains in place (FIFO1). When the next trigger is detected by the ESP32 and data is read, the data read is still from FIFO1, while the original data has been overwritten by the data packet number 5. Therefore, for the ESP32, it is equivalent to losing two frames of data packets with numbers 3 and 4.

[0048] The control terminal wirelessly receives image frame data packets, determines the image timing based on the frame number of the image frame data packets to avoid accidental frame loss, and verifies the data integrity of the local image data packets based on the packet numbers of all local image data packets in the image frame data packets.

[0049] The ESP32 and the control terminal use 2.4GHz Wi-Fi to transmit data via TCP. During data transmission, data packets may be lost due to factors such as channel contention and network interference. Frame number verification effectively eliminates incomplete and duplicate data frames. Since continuous data transmission is required and it's impossible to assign a unique identifier to each frame, an 8-bit frame number is used, cyclically marked within the range of 0-255.

[0050] Data packet loss mainly occurs in two places: one is during SPI communication between the FPGA and ESP32, which can easily lead to data loss; the other is during Wi-Fi transmission, where data loss can be caused by network interference. Packet number verification and frame number verification were performed for these two areas of data loss. One scenario is when the current frame is several frames behind the previous frame, indicating intermediate data loss. This type of frame loss often occurs because some frames preceding the current frame failed the initial verification, resulting in subsequent normal frames being several frames behind. Another scenario is when the current frame is identical to the previous frame. This is caused by the TCP retransmission mechanism and typically occurs after a period of Wi-Fi transmission interruption. During the data transmission interruption, the data in the ESP32's buffer that was not transmitted due to the interruption is overwritten by newly generated data. Therefore, when TCP retransmits, it can only send the data currently in the ESP32's buffer, resulting in the frame being sent twice. If this verification step is not performed to exclude duplicate frames, the duplicate data frames will be written into the video file. Since the video file is generated at a frame rate of 5 FPS, this could cause the recorded video to be longer than the actual recording time. Under normal circumstances, data duplication will not cause the loss of current data.

[0051] The specific verification method consists of two steps: Single-frame integrity check verifies whether the frame numbers in a whole frame of transmitted image are consistent. Since we have already labeled each packet in the image with a frame number during the initial labeling process, it is easy to determine whether a frame of data received by the control terminal comes from the same frame. If it does not come from the same frame, the frame is discarded and does not proceed to the timing integrity check.

[0052] The timing integrity check, assuming the single-frame integrity check passes, compares the frame number with the previous frame received by the control terminal to check for any dropped frames. If a missing frame number is found, it is considered a dropped frame. If a dropped frame is found, an error will be reported to the user, and the dropped frame information will be recorded for easy user retrieval.

[0053] Through these two verification steps, the data that passes the verification will be displayed and stored. Although the two verification steps cannot recover lost data frames, they can ensure that every frame of the displayed and stored data is complete. Moreover, if there are any lost frames, users can obtain the lost frame information through timing integrity verification, avoiding accidental frame loss that could lead to timing misalignment.

[0054] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A dual processing chip wearable microscope system based on WIFI image transmission, characterized in that, The image sensor, the lower image acquisition chip, and the upper image acquisition chip are included. The image sensor is configured to sense a light signal and generate raw image data; the raw image data is an array of pixel points in digital form. The lower image acquisition chip is configured to read the raw image data, cache and package the raw image data into local image data packets, and forward the local image data packets to the upper image acquisition chip; the local image data packet includes an array of pixel points in digital form of a preset region of the raw image data, a frame number, and a packet number; the packet number is used to represent the preset region of the raw image data where the data packet is located, and has a start packet number and an end packet number in sequence; and the frame number is used to mark the raw image where the data packet is located. The upper image acquisition chip is configured to check the data integrity of the local image data packet, package the local image data packet into an image frame data packet, and send the image frame data packet to a control terminal using WIFI image transmission technology. The control terminal wirelessly receives the image frame data packet, and determines the image timing according to the frame number of the image frame data packet to avoid accidental frame loss.

2. The dual processing chip WIFI-based image transmission wearable microscope system of claim 1, wherein, The lower image acquisition chip caches the local image data of the preset region of the raw image to package the local image data into local image data packets; and preferably, the lower image acquisition chip provides multiple cache queues to acquire the raw image.

3. The dual processing chip WIFI-based image transmission wearable microscope system of claim 2, wherein, The lower image acquisition chip replaces the edge pixel points with the frame number and / or the packet number.

4. The dual processing chip WIFI-based image transmission wearable microscope system of claim 1, wherein, The lower image acquisition chip directly transmits data with the image sensor; and the lower image acquisition chip and the upper image acquisition chip preferably communicate using the SPI protocol and receive imaging control signals.

5. The dual processing chip WIFI-based image transmission wearable microscope system of claim 1, wherein, The image frame data packet includes all local image data packets of a frame of raw image data; and the data integrity of the local image data packet is checked by determining that the image frame data is composed of all local image data packets of a frame of raw image data according to the frame number and the packet number.

6. The dual processing chip WIFI-based image transmission wearable microscope system of claim 1, wherein, The lower image acquisition chip is used as the main transmission chip, and a handshake mechanism is used to establish a channel for generating a control clock by the upper image acquisition chip to cache and forward the local image data packet in units of image frames, as follows: After the lower image acquisition chip completes caching and packaging of a new local image data packet, the lower image acquisition chip sends a handshake signal to the upper image acquisition chip; The upper image acquisition chip establishes a data communication channel and generates a periodic transmission clock according to the handshake signal; The transmission time length of the transmission clock is less than the cache time length of the lower image acquisition chip for generating the local image data packet; The lower image acquisition chip sends all local image data packets of the frame of image according to the transmission clock generated by the upper image acquisition chip; After receiving a preset number of local image data packets, the upper image acquisition chip checks the packet number, ensures the data integrity of the local image data packet of the frame of image, packages all local image data packets of the frame of image into an image frame data packet, and sends the image frame data packet to the control terminal using WIFI image transmission technology.

7. A method for transmitting image data double-checked, characterized by, The method includes the following steps: (1) The lower image acquisition chip divides each frame of original image data into local images and packs them into local image data packets, which are sent to the upper image acquisition chip; the local image data packet includes a pixel point array of the digital representation of the original image data in a preset area, a frame number, and a packet number; the packet number is used to represent the original image preset area where the data packet is located, and has a starting packet number and a terminating packet number in sequence; the frame number is used to mark the original image where the data packet is located; (2) After the upper image acquisition chip receives the local image data packet, it analyzes the frame number and packet number of the data packet; if the packet number of the local image data packet is the same as the current expected packet number, the data is cached, otherwise the local image data packet is discarded, the cache is emptied, and the expected packet number is reset to the starting packet number; when the packet number of the cached local image data packet is the terminating packet number, the cached local image data packet is packed into an image frame data packet and forwarded to the control terminal; (3) After the control terminal receives the image frame data packet, it analyzes the frame number of the data packet and compares it with the frame number of the received image frame data packet to check if there is a missing frame number or a duplicate frame number; when there is a missing frame number, it is judged as a lost frame and the missing frame number is recorded; when there is a duplicate frame number, it is judged as a data retransmission and the image frame data packet is discarded.

8. The method of claim 7, wherein the image data is transmitted in a dual- checked manner. In step (1), the lower image acquisition chip uses a handshaking mechanism to establish a channel for generating a control clock according to the upper image acquisition chip to cache and forward the local image data packet in units of image frames; the specific process is as follows: After the lower image acquisition chip completes the caching and packing of a new local image data packet, it sends a handshake signal to the upper image acquisition chip; The upper image acquisition chip establishes a data communication channel and generates a periodic transmission clock according to the handshake signal; The transmission time of the transmission clock is less than the caching time of the local image data packet generated by the lower machine; The lower image acquisition chip sends all local image data packets of the frame image according to the transmission clock generated by the upper image acquisition chip.

9. The method of claim 7, wherein the image data is transmitted in a dual- checked manner. In step (2), the upper image acquisition chip judges whether the frame numbers of all local image data packets in the cache data are the same; if they are the same, the cached local image data packets are packed into an image frame data packet, otherwise the cache is emptied and the expected packet number is terminated as the starting packet number.

10. The method of claim 8, wherein the image data is transmitted in a dual- checked manner. In step (3), the upper image acquisition chip performs single-frame integrity verification to judge whether the frame numbers of all local image data packets in the cache data are the same; if they are the same, the cached local image data packets are packed into an image frame data packet, otherwise the cache is emptied and the expected packet number is terminated as the starting packet number.

Images