Sensor array data transmission method and apparatus

Abstract

The application provides a sensor array data transmission method applied to an FPGA, wherein the FPGA has a PDM digital interface and a DVP digital interface; a multi-channel PDM signal stream output by a digital sensor array is received through the PDM digital interface; the multi-channel PDM signal stream is converted into a multi-channel PCM data stream by using a synchronous digital modulation mode; and a dynamic fragmentation compression process is performed on the multi-channel PCM data stream by using an equal-probability sampling method; a video data stream output by an image sensor is received through the DVP digital interface; the video data stream is compressed by using a H264-like video data compression method; the multi-channel PCM data stream is combined and encoded; the encoded PCM data stream is inserted into the compressed video data stream by using an interpolation method; the encoded PCM data stream and the compressed video data stream are kept time-synchronous; and the mixed data stream is packaged and uploaded to a cloud server through a gigabit network communication protocol.

Description

Technical Field

[0001] This invention relates to the field of data transmission, and more specifically, to a method and apparatus for data transmission of a sensor array. Background Technology

[0002] In industries such as petrochemicals, power, and automobiles, the inspection and maintenance of pipelines and equipment has always been a crucial task. Regular inspections by personnel are essential for timely detection of abnormal conditions such as gas leaks, abnormal discharges, or mechanical vibrations, which is of great significance for ensuring production, as well as the safety of personnel and property.

[0003] Traditional inspection methods rely on manual labor, with inspectors using methods such as listening, visual observation, and checking for bubbles in soapy water to inspect and maintain equipment and pipelines for leaks and other anomalies. This approach is highly dangerous, inefficient, and prone to missing detections. Research has shown that gas leaks, abnormal discharges, and mechanical vibrations all generate sound, leading to the development of acoustic imaging for sound source localization. Acoustic imaging utilizes high-sensitivity digital MEMS microphones and array signal processing algorithms to generate sound pressure level distributions on a plane. This is visualized as a color contour map, and the sound distribution of the measured object can be intuitively displayed through photos or videos. This method effectively compensates for the limitations of human hearing in locating sound sources and offers a more efficient solution for industrial detection. For example, CN117028870A provides an acoustic imaging system for locating natural gas leaks based on ultrasonic arrays and images.

[0004] However, existing ultrasonic imaging detection still faces significant challenges in terms of accuracy, sensitivity, resolution, and wide-area imaging capabilities. This is because existing ultrasonic arrays use analog acquisition methods, employing complex ADC conversion circuits to convert acoustic signals; the unit size is large, making it impossible to achieve more channels within the same volume; ensuring consistent accuracy across multiple channels is difficult; and overall power consumption and maintenance / production costs are increased. Furthermore, existing ultrasonic arrays use narrowband communication methods such as USB or serial bus communication, which are limited by communication speed, preventing the achievement of higher sampling rates; they also cannot perform continuous acquisition, real-time online operation, or distributed system deployment.

[0005] In addition, with the development of 5G network technology, it has become possible to realize the remote transmission and real-time analysis of ultrasound images. However, how to efficiently integrate ultrasound array technology with network communication while ensuring imaging quality and system response speed still faces technical challenges.

[0006] In order to solve the above problems, people have been seeking an ideal technological solution. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of existing technologies by providing a sensor array data transmission method, apparatus, and ultrasonic array acoustic image acquisition device.

[0008] To achieve one of the above objectives, the technical solution adopted by the present invention is: a sensor array data transmission method applied to an FPGA, wherein the FPGA has a PDM digital interface and a DVP digital interface, and the method includes:

[0009] The PDM digital interface receives the multi-channel PDM signal stream output by the digital sensor array, and uses synchronous digital modulation to convert the multi-channel PDM signal stream into a multi-channel PCM data stream. The multi-channel PCM data stream is then dynamically segmented and compressed using a random sampling method.

[0010] The video data stream output by the image sensor is received through the DVP digital interface, and the video data stream is compressed using an H264-like video data compression method.

[0011] Multi-channel PCM data streams are combined and encoded, and the encoded PCM data streams are interpolated into the compressed video data streams. The encoded PCM data streams and the compressed video data streams are kept in time synchronization.

[0012] The mixed data streams are packaged and uploaded to the cloud server using gigabit network communication protocols.

[0013] Furthermore, before performing dynamic fragmentation and compression processing on the multi-channel PCM data stream using random sampling, asynchronous FIFO buffering is also performed on the multi-channel PCM data stream to ensure the continuity of the multi-channel PCM data stream.

[0014] Before compressing the video data stream using an H264-like video data compression method, the video data stream is also subjected to asynchronous FIFO buffering to ensure the continuity of the video data stream.

[0015] Furthermore, the asynchronous FIFO buffering process includes the following steps:

[0016] Upon receiving the frame synchronization signal from the image sensor, reset the input FIFO;

[0017] Input data is received at the first sampling frequency and written into the input FIFO. When the amount of buffered data in the input FIFO exceeds the first preset warning threshold, the buffered data in the input FIFO is written into a storage bank in the DDR controller at the second sampling frequency. After writing a complete frame of data, a frame synchronization signal is sent to the DDR controller to perform a storage bank switching action.

[0018] After receiving the frame synchronization signal, the DDR controller performs a bank switching operation during the gap between when one bank completes data reception and another bank completes data transmission. This connects the bank that has completed data reception to the output FIFO and the bank that has completed data transmission to the input FIFO, while simultaneously resetting the output FIFO.

[0019] After the output FIFO is reset, cached data is read from the DDR controller's storage bank and written to it so that external data can be read at the third sampling frequency; and when the amount of written data is less than the second warning threshold, cached data is continuously read from the DDR controller's storage bank.

[0020] Furthermore, the specific steps for compressing video data streams using an H.264-like video data compression method include:

[0021] Each frame of video image in the video data stream is binarized to obtain a grayscale image;

[0022] Motion estimation is performed on the current frame video image based on the reference frame video image to obtain the motion vector and motion residual matrix;

[0023] The motion residual matrix is ​​quantized, and the pixels in the current frame video image are divided into intra-frame variation, fixed amount and motion vector based on the quantized motion residual matrix.

[0024] Intra-frame compression is performed on pixels in the current frame video image by sorting by difference. The compressed pixels are then encoded according to a specific encoding format to obtain the frame compressed data of the current frame video image. For variable values, pixels with consistent colors are arranged and compressed, while pixels with changing colors are categorized and compressed. For fixed values ​​and motion vectors, compression is performed by transmitting only the reference frame ID and reference position.

[0025] Based on the principle of frame coding, each frame of data is encapsulated according to timestamp, reference frame ID, difference type, difference vector, and the format of frame compressed data;

[0026] After performing difference sorting, the background is stripped from the change amount and / or motion vector, and the current frame video image is used as the reference frame for the next frame video image.

[0027] A second objective of this invention is to provide a sensor array data transmission device, including an FPGA, wherein the FPGA has a PDM digital interface and a DVP digital interface; the PDM digital interface is connected to a digital sensor array and is used to receive a multi-channel PDM signal stream output by the digital sensor array; the DVP digital interface is connected to an image sensor and is used to receive a video data stream.

[0028] The FPGA is also equipped with a signal processor, an image processor, a mixer, and a network data processing unit.

[0029] The signal processor includes a synchronous digital modulation module and a dynamic data segmentation and compression module. The synchronous digital modulation module is used to convert a multi-channel PDM signal stream into a multi-channel PCM data stream using synchronous digital modulation. The dynamic data segmentation and compression module is used to perform dynamic segmentation and compression processing on the multi-channel PCM data stream using a random sampling method.

[0030] The image processor includes a video data compression module, used to compress the video data stream using an H.264-like video data compression method;

[0031] The mixer is used to combine and encode multi-channel PCM data streams, and interpolate the encoded PCM data streams into the compressed video data streams using interpolation, wherein the encoded PCM data streams and the compressed video data streams are time-synchronized.

[0032] The network data processing unit is used to package and upload the interleaved and mixed data stream to the cloud server via gigabit network communication protocol.

[0033] Furthermore, the signal processor also includes a first asynchronous buffer module, which is disposed between the synchronous digital modulation module and the data dynamic segmentation and compression module, and is used to perform asynchronous FIFO buffering processing on the multi-channel PCM data stream to achieve the continuity of the multi-channel PCM data stream;

[0034] The image processing unit further includes a second asynchronous cache module, which is located before the video data compression module and is used to perform asynchronous FIFO caching processing on the video data stream to achieve the continuity of the video data stream.

[0035] Furthermore, the video data compression module includes:

[0036] The binarization sub-block is used to binarize each frame of video image in the video data stream to obtain a grayscale image.

[0037] Motion change acquisition sub-block is used to perform motion estimation on the current frame video image based on the reference frame video image, and obtain motion vectors and motion residual matrices;

[0038] Intra-frame data is divided into sub-blocks to quantize the motion residual matrix, and the pixels in the current frame video image are divided into intra-frame change quantities, fixed quantities, and motion vectors based on the quantized motion residual matrix.

[0039] The compression sub-block is used to perform intra-frame compression on pixels in the current frame video image by sorting by difference. The compressed pixels are then encoded according to a specific encoding format to obtain the frame compression data of the current frame video image. For variable values, pixels with consistent colors are arranged and compressed, while pixels with changing colors are categorized and compressed. For fixed values ​​and motion vectors, compression is performed by transmitting only the reference frame ID and reference position.

[0040] The bitstream formatting module is used to encapsulate each frame of data according to the principles of frame encoding, including timestamp, reference frame ID, difference type, difference vector, and frame compression data format.

[0041] The reference frame update module is used to strip the background from the change and / or motion vector after performing difference sorting and to use the current frame video image as the reference frame for the next frame video image.

[0042] The third objective of this invention is to provide a sensor array data acquisition and transmission device, including a digital sensor array, an image sensor, and the aforementioned sensor array data transmission device;

[0043] The digital sensor array uses a non-uniform oscillating element arrangement to form a multi-channel signal acquisition unit, which is used to acquire and output multi-channel PDM signal streams.

[0044] The image sensor is positioned at the center of the digital sensor array and is used to acquire and output video data streams;

[0045] The sensor array data transmission device processes multi-channel PDM signal streams and video image data to form a mixed data stream, which is then uploaded to the cloud server.

[0046] Furthermore, the digital sensor array includes an ultrasonic array sensor, a microphone array, an infrared digital sensor array, or a laser digital sensor array.

[0047] Furthermore, non-uniform oscillator arrangement methods include rectangular arrays, central arrays, multi-arm spiral arrays, irregular arrays, and ring or polyhedral arrays.

[0048] Compared with the prior art, the present invention has outstanding substantive features and significant progress. Specifically, the present invention uses an integrated data transmission method to simultaneously process the multi-channel PDM signal stream output by the digital sensor array and the video data stream output by the image sensor, and send them out from the same interface, thereby reducing the difficulty of subsequent processing and improving the flexibility of subsequent data processing.

[0049] Based on digital design, FPGA integration and algorithm optimization, we achieve high sampling rate and more accurate signal analysis and feature extraction, so as to eliminate external environmental interference and more accurately determine the source of abnormal sound.

[0050] The combination of digital sensor arrays and synchronous digital modulation for data processing reduces sampling circuitry compared to analog processing, resulting in a more streamlined overall device BOM and significantly smaller size. Interference from external environmental signals is also greatly reduced during analog signal processing, lowering data source distortion and further ensuring the integrity of the output results. It also allows for greater flexibility in subsequent digital algorithm processing.

[0051] Data is uploaded via gigabit network communication protocol, supporting higher sampling rate sensor signal acquisition and higher resolution image data. It has good compatibility and can be directly connected to the Internet, enabling remote control across regions. Attached Figure Description

[0052] Figure 1 This is a flowchart of the transmission method described in Embodiment 1 of the present invention.

[0053] Figure 2 This is a flowchart illustrating the synchronous digital modulation method described in Embodiment 1 of the present invention.

[0054] Figure 3 This is a schematic diagram of the H.264-like video data compression method described in Embodiment 1 of the present invention.

[0055] Figure 4 This is a schematic diagram of pixel encoding in Embodiment 1 of the present invention.

[0056] Figure 5 This is a schematic diagram of frame encoding in Embodiment 1 of the present invention.

[0057] Figure 6 This refers to the change in bandwidth occupied by the data stream before and after compression of H264-like video data in Embodiment 1 of the present invention.

[0058] Figure 7 This is the interpolation reference timing in Embodiment 1 of the present invention.

[0059] Figure 8 This is a flowchart of the transmission method described in Embodiment 2 of the present invention.

[0060] Figure 9 This is a schematic diagram of the asynchronous FIFO caching process described in Embodiment 2 of the present invention.

[0061] Figure 10 This is a schematic diagram of the sensor array data transmission device described in Embodiment 3 of the present invention.

[0062] Figure 11 This is a schematic diagram of the sensor array data acquisition and transmission device described in Embodiment 4 of the present invention.

[0063] Figure 12This is the method of the sensor array data acquisition and transmission device described in Embodiment 4 of the present invention.

[0064] In the diagram: 1. Digital sensor array; 2. Image sensor. Detailed Implementation

[0065] The technical solution of the present invention will be further described in detail below through specific embodiments.

[0066] Example 1

[0067] This embodiment provides a sensor array data transmission method applied to an FPGA, wherein the FPGA has a PDM digital interface and a DVP digital interface, such as Figure 1 As shown, the method includes:

[0068] The PDM digital interface receives the multi-channel PDM signal stream output by the digital sensor array 1, and uses synchronous digital modulation to convert the multi-channel PDM signal stream into a multi-channel PCM data stream. The multi-channel PCM data stream is then dynamically segmented and compressed using a random sampling method.

[0069] Digital sensor array 1 outputs a PDM carrier signal. PDM signal is a modulation method that uses digital signals to represent analog signals, i.e., a frequency carrier signal. The advantage of PDM is that it can highly reproduce real analog signals and reduce signal distortion. However, PDM is essentially an analog signal and cannot be directly recognized by upper-level algorithms, so it needs to be converted into a digital PCM signal.

[0070] This embodiment uses a synchronous digital modulation algorithm to convert the PDM signal into a digital PCM signal. Specifically, the synchronous digital modulation algorithm is implemented based on an FPGA and reconstructs the analog signal through multiple integrations and samplings. For example... Figure 2 As shown, it includes a first-order integrator, a second-order integrator, a decimation / downsampling module, a first-order comb, and a second-order comb.

[0071] Specifically, random sampling methods include simple random sampling, equal probability sampling, stratified sampling, and cluster sampling. The appropriate method can be selected based on the specific application. For example, if the upper-layer application needs to process real-time audio variation patterns for time-domain analysis, equal probability sampling should be used; while if the upper-layer application only performs feature processing and does not perform time-domain analysis, other random sampling methods can be used.

[0072] The video data stream output from image sensor 2 is received via a DVP digital interface, and compressed using an H.264-like video data compression method. Preferably, image sensor 2 is a camera.

[0073] In one embodiment, such as Figure 3 As shown, the specific steps for compressing a video data stream using an H.264-like video data compression method include:

[0074] Each frame of video image in the video data stream is binarized to obtain a grayscale image; the binarized image has a more uniform color depth, which is more conducive to subsequent data processing.

[0075] Motion estimation is performed on the current frame video image based on the reference frame video image to obtain the motion vector and motion residual matrix. Motion estimation is the process of finding the optimal or suboptimal motion vector when the current frame image has undergone motion changes relative to the reference frame image. Motion change means that the color of the current frame image and the reference frame image has changed at a certain pixel. Specifically, motion changes are divided into three categories: local change, overall movement, and full dynamics. However, since the application scenarios of this embodiment are mostly fixed monitoring, only local change and overall movement need to be considered.

[0076] It should be noted that the reference frame changes in real time as the frame moves, and generally the previous video frame is used as the reference frame for the next video frame; the size of the motion vector affects the data compression ratio. If the variable is small, the compression ratio is high, and vice versa.

[0077] The motion residual matrix is ​​quantized, and the pixels in the current frame video image are divided into intra-frame variation, fixed amount and motion vector based on the quantized motion residual matrix.

[0078] Intra-frame compression is performed on pixels in the current frame of the video image by sorting by difference. The compressed pixels are then encoded according to a specific encoding format to obtain the frame compressed data of the current frame of the video image. For variables, pixels with consistent color are compressed in an orderly manner, while pixels with changing color are categorized and compressed. For fixed variables and motion vectors, compression is performed by transmitting only the reference frame ID and reference position. Figure 4 The image shows an example of intra-frame compression and pixel coding.

[0079] It is understandable that encoding corresponds to decoding, so the basic requirement for encoding is that it should be easy to decode in order to improve efficiency.

[0080] Based on the principles of frame coding, each frame of data is encapsulated according to its timestamp, reference frame ID, difference type, difference vector, and the format of the compressed frame data. Specifically, as follows... Figure 5 As shown.

[0081] like Figure 6 As shown, this compares the changes in bandwidth usage of the data stream before and after data compression.

[0082] As can be seen, the actual image quality is within an acceptable range, with the frame rate controlled at 15fps and each frame image size being approximately 14KB.

[0083] Furthermore, compared to the standard H.264 inter-frame predictive compression algorithm, which predicts and judges the original colors of the image and can highly restore the real image changes, while also being excellent in terms of image distortion and color reproduction; however, its disadvantage is that it has high requirements for computing power and hardware. The advantages and disadvantages of the H.264-like video data compression method in this embodiment are also very obvious. Due to the use of grayscale quantization, color accuracy is lost, and compression is limited to inter-frame movement differences. Its advantages are: it greatly reduces the data volume of the image stream; its disadvantages are: poor compression effect on fast-moving targets and relatively poor image reproduction. However, since this embodiment uses directional monitoring and fixed camera position monitoring, it does not affect the compression effect.

[0084] Understandably, after performing the difference sorting, the background is stripped from the change and / or motion vectors, and the current frame video image is used as the reference frame for the next frame video image.

[0085] After compressing both the multi-channel PCM data stream and the video data stream, since the PDM signal is multi-channel, the multi-channel PCM data stream needs to be combined and encoded. Furthermore, after combining and encoding the multi-channel PCM data stream, the encoded PCM data stream is interpolated into the compressed video data stream using interpolation, ensuring that the encoded PCM data stream and the compressed video data stream are time-synchronized.

[0086] Under normal circumstances, multi-channel PCM data streams consume less bandwidth, while video data streams consume more. Therefore, when both share network bandwidth, to ensure video and audio synchronization, the multi-channel PCM data stream needs to be interspersed between the video data streams, while maintaining time synchronization between them. For specific instructions, please refer to [reference needed]. Figure 7 The timing sequence is shown. Specifically, the number of audio frames inserted can be dynamically adjusted based on the real-time data volume.

[0087] Finally, the interleaved and mixed data streams are packaged and uploaded to the cloud server via gigabit network communication protocols.

[0088] It is important to note that before uploading the interleaved and mixed data stream to the cloud server via gigabit network communication protocol, you need to configure the IP address and server address on the device and establish an ARP connection with the server. After preparation, wait to receive the interleaved and mixed data stream.

[0089] As can be seen, this embodiment uses an integrated data transmission method to process the multi-channel PDM signal stream output by the digital sensor array 1 and the video data stream output by the image sensor 2 simultaneously, and send them out from the same interface, reducing the difficulty of subsequent processing and improving the flexibility of subsequent data processing.

[0090] Based on digital design, FPGA integration, and algorithm optimization, a high sampling rate and more accurate signal analysis and feature extraction are achieved to eliminate external environmental interference and more accurately determine the source of abnormal sound.

[0091] Data is uploaded via gigabit network communication protocol, supporting higher sampling rate sensor signal acquisition and higher resolution image data. It has good compatibility and can be directly connected to the Internet, enabling remote control across regions.

[0092] Example 2

[0093] This embodiment provides another implementation of the sensor array data transmission method.

[0094] In this embodiment, such as Figure 8 As shown, before performing dynamic fragmentation and compression processing on the multi-channel PCM data stream using random sampling, asynchronous FIFO buffering is also performed on the multi-channel PCM data stream to ensure the continuity of the multi-channel PCM data stream.

[0095] Before compressing the video data stream using an H264-like video data compression method, the video data stream is also subjected to asynchronous FIFO buffering to ensure the continuity of the video data stream.

[0096] Specifically, asynchronous FIFO buffering is performed on the multi-channel PCM data stream to achieve 1024-bit input and 32-bit output; asynchronous FIFO buffering is also performed on the video data stream to achieve 32-bit input and 256-bit output.

[0097] The steps of asynchronous FIFO buffering are the same. For ease of understanding, asynchronous FIFO buffering of video data streams will be used as an example. See [link to relevant documentation]. Figure 9 The steps of the asynchronous FIFO buffering process are described in detail, including:

[0098] Upon receiving the frame synchronization signal from image sensor 2, reset the input FIFO;

[0099] Input data is received at a first sampling frequency (e.g., a sampling clock of 50MHz) and written into an input FIFO. When the amount of buffered data in the input FIFO exceeds a first preset warning threshold, the buffered data in the input FIFO is triggered to be written into a storage bank in the DDR controller at a second sampling frequency (e.g., a sampling clock of 100MHz). After writing a complete frame of data, a frame synchronization signal is sent to the DDR controller to perform a storage bank switching action. Generally, the first preset warning threshold is not greater than 1 / 2 of the rated buffer size of the input FIFO.

[0100] After receiving the frame synchronization signal, the DDR controller performs a storage bank switching operation during the interval between data reception in one storage bank and data transmission in another. This connects the storage bank that has completed data reception to the output FIFO, and the storage bank that has completed data transmission to the input FIFO, while simultaneously resetting the output FIFO. It can be understood that the DDR controller has at least two storage banks, sufficient to meet scheduling needs within one frame transmission / reception cycle. The purpose of setting two storage banks is to ensure frame integrity during data stream transitions at different frequencies, while also ensuring the streaming characteristics of the frame data. It is important to note that the current input frame and output frame cannot be in the same storage bank.

[0101] After the output FIFO is reset, cached data is read from the DDR controller's storage bank and written to it for external data reading at a third sampling frequency (e.g., a sampling clock of 125MHz). Furthermore, when the amount of data written is less than a second warning threshold, cached data is continuously read from the DDR controller's storage bank. The second warning threshold is not less than half the rated cache size of the output FIFO.

[0102] It is important to note that within the valid frame interval, the output FIFO must always be kept active. When the output FIFO contains data, the sending module must schedule the sending mechanism as soon as possible to send the data out.

[0103] The above asynchronous buffering process is implemented by input / output FIFOs and FPGA internal BRAM, with each buffer having a depth of 2K; while the frame buffer needs to meet the requirements of a complete frame, so it is implemented using external DDR.

[0104] Example 3

[0105] This embodiment provides a sensor array data transmission device, such as... Figure 10As shown, the system includes an FPGA, which has a PDM digital interface and a DVP digital interface; the PDM digital interface is connected to a digital sensor array 1 and is used to receive a multi-channel PDM signal stream output by the digital sensor array 1; the DVP digital interface is connected to an image sensor 2 and is used to receive a video data stream.

[0106] The FPGA is also equipped with a signal processor, an image processor, a mixer, and a network data processing unit.

[0107] The signal processor includes a synchronous digital modulation module and a dynamic data segmentation and compression module. The synchronous digital modulation module is used to convert a multi-channel PDM signal stream into a multi-channel PCM data stream using synchronous digital modulation. The dynamic data segmentation and compression module is used to perform dynamic segmentation and compression processing on the multi-channel PCM data stream using a random sampling method.

[0108] The image processor includes a video data compression module, used to compress the video data stream using an H.264-like video data compression method;

[0109] The mixer is used to combine and encode multi-channel PCM data streams, and interpolate the encoded PCM data streams into the compressed video data streams using interpolation, wherein the encoded PCM data streams and the compressed video data streams are time-synchronized.

[0110] The network data processing unit is used to package and upload the interleaved and mixed data stream to the cloud server via gigabit network communication protocol.

[0111] Furthermore, the signal processor also includes a first asynchronous buffer module, which is disposed between the synchronous digital modulation module and the data dynamic segmentation and compression module, and is used to perform asynchronous FIFO buffering processing on the multi-channel PCM data stream to achieve the continuity of the multi-channel PCM data stream;

[0112] The image processing unit further includes a second asynchronous cache module, which is located before the video data compression module and is used to perform asynchronous FIFO caching processing on the video data stream to achieve the continuity of the video data stream.

[0113] Specifically, such as Figure 9 As shown, the asynchronous FIFO cache processing steps include:

[0114] Upon receiving the frame synchronization signal from image sensor 2, reset the input FIFO;

[0115] Input data is received at a first sampling frequency (e.g., a sampling clock of 50MHz) and written into an input FIFO. When the amount of buffered data in the input FIFO exceeds a first preset warning threshold, the buffered data in the input FIFO is triggered to be written into a storage bank in the DDR controller at a second sampling frequency (e.g., a sampling clock of 100MHz). After writing a complete frame of data, a frame synchronization signal is sent to the DDR controller to perform a storage bank switching action. Generally, the first preset warning threshold is not greater than 1 / 2 of the rated buffer size of the input FIFO.

[0116] After receiving the frame synchronization signal, the DDR controller performs a storage bank switching operation during the interval between data reception in one storage bank and data transmission in another. This connects the storage bank that has completed data reception to the output FIFO, and the storage bank that has completed data transmission to the input FIFO, while simultaneously resetting the output FIFO. It can be understood that the DDR controller has at least two storage banks, sufficient to meet scheduling needs within one frame transmission / reception cycle. The purpose of setting two storage banks is to ensure frame integrity during data stream transitions at different frequencies, while also ensuring the streaming characteristics of the frame data. It is important to note that the current input frame and output frame cannot be in the same storage bank.

[0117] After the output FIFO is reset, cached data is read from the DDR controller's storage bank and written to it for external data reading at a third sampling frequency (e.g., a sampling clock of 125MHz). Furthermore, when the amount of data written is less than a second warning threshold, cached data is continuously read from the DDR controller's storage bank. The second warning threshold is not less than half the rated cache size of the output FIFO.

[0118] It is important to note that within the valid frame interval, the output FIFO must always be kept active. When the output FIFO contains data, the sending module must schedule the sending mechanism as soon as possible to send the data out.

[0119] Furthermore, the video data compression module includes:

[0120] The binarization sub-block is used to binarize each frame of video image in the video data stream to obtain a grayscale image.

[0121] Motion change acquisition sub-block is used to perform motion estimation on the current frame video image based on the reference frame video image, and obtain motion vectors and motion residual matrices;

[0122] Intra-frame data is divided into sub-blocks to quantize the motion residual matrix, and the pixels in the current frame video image are divided into intra-frame change quantities, fixed quantities, and motion vectors based on the quantized motion residual matrix.

[0123] The compression sub-block is used to perform intra-frame compression on pixels in the current frame video image by sorting by difference. The compressed pixels are then encoded according to a specific encoding format to obtain the frame compression data of the current frame video image. For variable values, pixels with consistent colors are arranged and compressed, while pixels with changing colors are categorized and compressed. For fixed values ​​and motion vectors, compression is performed by transmitting only the reference frame ID and reference position.

[0124] The bitstream formatting module is used to encapsulate each frame of data according to the principles of frame encoding, including timestamp, reference frame ID, difference type, difference vector, and frame compression data format.

[0125] The reference frame update module is used to strip the background from the change and / or motion vector after performing difference sorting and to use the current frame video image as the reference frame for the next frame video image.

[0126] Example 4

[0127] This embodiment provides a sensor array data acquisition and transmission device, such as... Figure 11 As shown, it specifically includes a digital sensor array 1, an image sensor 2, and the sensor array data transmission device described in Embodiment 3. It can be understood that it also includes a power supply unit. The power supply unit integrates multiple power supplies to ensure the stable and reliable operation of the digital sensor array 1, processing unit, image acquisition, data transmission and other functions; and provides necessary abnormal handling mechanisms to effectively and promptly cut off the power supply in the event of overheating, overload and other conditions to ensure equipment safety.

[0128] The digital sensor array 1 includes an ultrasonic array sensor, a microphone array, an infrared digital sensor array 1, or a laser digital sensor array 1. Specifically, regardless of the type of digital sensor array 1, it outputs sensor signals in PDM format.

[0129] This embodiment takes an ultrasonic array sensor as an example. The ultrasonic array sensor includes multiple high-sensitivity ultrasonic sensors, generally microphones. Multiple high-sensitivity ultrasonic sensors are arranged in a specific array to form a multi-channel ultrasonic data acquisition unit to collect multi-channel data. This data is processed by a specific algorithm, such as DOA (beamforming algorithm), to accurately obtain feature data with directional attributes, which can be used for sound source localization.

[0130] Furthermore, non-uniform oscillator arrangement methods include rectangular arrays, central arrays, multi-arm spiral arrays, irregular arrays, and ring or polyhedral arrays. In practice, array layouts are not limited to the above, and different arrangements can be applied to different application scenarios.

[0131] Multi-channel array element layouts can acquire data with at least 16 channels and up to 512 channels, increasing acquisition accuracy while enabling wide-angle, multi-phase data acquisition. In practice, the number of channels is not limited, but 64 channels are common. Different application scenarios have different requirements for data accuracy. In general, increasing the number of channels can increase the number of samples and improve the accuracy of the output results.

[0132] The image sensor 2 is integrated with the digital sensor array 1. Specifically, the image sensor 2 is located at the center of the digital sensor array 1, so that the image sensor 2 and the digital sensor array 1 are in the same vector direction, which facilitates the organic fusion of different types of data.

[0133] In practical implementation, the image sensor 2 uses a high-definition camera to achieve high-definition image acquisition and high-definition video recording, thereby improving audio-visual clarity and reducing artifacts, ghosting, and other phenomena.

[0134] The sensor array data transmission device includes an FPGA, which has a PDM digital interface and a DVP digital interface; the PDM digital interface is connected to the digital sensor array 1 and is used to receive the multi-channel PDM signal stream output by the digital sensor array 1; the DVP digital interface is connected to the image sensor 2 and is used to receive the video data stream.

[0135] The FPGA is also equipped with a signal processor, an image processor, a mixer, and a network data processing unit.

[0136] The signal processor includes a synchronous digital modulation module and a dynamic data segmentation and compression module. The synchronous digital modulation module is used to convert a multi-channel PDM signal stream into a multi-channel PCM data stream using synchronous digital modulation. The dynamic data segmentation and compression module is used to perform dynamic segmentation and compression processing on the multi-channel PCM data stream using a random sampling method.

[0137] The image processor includes a video data compression module, used to compress the video data stream using an H.264-like video data compression method;

[0138] The mixer is used to combine and encode multi-channel PCM data streams, and interpolate the encoded PCM data streams into the compressed video data streams using interpolation, wherein the encoded PCM data streams and the compressed video data streams are time-synchronized.

[0139] The network data processing unit is used to package and upload the interleaved and mixed data stream to the cloud server via gigabit network communication protocol.

[0140] The network data processing unit is used to package and send the ultrasound video stream.

[0141] Specifically, such as Figure 12 As shown, the working principle of an ultrasonic array sensor, taking it as an example, is as follows:

[0142] The ultrasonic array detector acquires raw digital ultrasonic data from multiple channels; the sensor processing unit performs logical processing and filtering on the multi-channel ultrasonic data to restore the ultrasonic data stream.

[0143] The image sensor 2 acquires image data; the image processing unit performs logical processing and noise reduction on the image data to restore high-definition image data.

[0144] The high-speed data fusion unit is used to merge the ultrasound data stream and high-definition image data to form an ultrasound video stream, which is then packaged and sent out by the network data processing unit.

[0145] In this process, asynchronous FIFO buffering is performed after obtaining both the ultrasound data stream and the video data stream to ensure the continuity of the data stream. Specifically, for example... Figure 8 As shown, asynchronous FIFO buffering is performed on the multi-channel PCM data stream to achieve 1024-bit input and 32-bit output; asynchronous FIFO buffering is also performed on the video data stream to achieve 32-bit input and 256-bit output.

[0146] Before uploading the interleaved and mixed data stream to the cloud server via gigabit network communication protocol, it is necessary to configure the IP address and server address on the device and establish an ARP connection with the server; after preparation, wait to receive the interleaved and mixed data stream.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them; although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications can still be made to the specific implementation of the present invention or equivalent substitutions can be made to some technical features without departing from the spirit of the technical solutions of the present invention, and all such modifications and substitutions should be covered within the scope of the technical solutions claimed in the present invention.

Claims

1. A sensor array data transmission method, applied to an FPGA, wherein the FPGA has a PDM digital interface and a DVP digital interface, characterized in that: The system receives multi-channel PDM signal streams from a digital sensor array via a PDM digital interface. It then converts these PDM signal streams into multi-channel PCM data streams using synchronous digital modulation. Asynchronous FIFO buffering is applied to the multi-channel PCM data streams to ensure their continuity. Dynamic segmentation and compression are performed on the multi-channel PCM data streams using random sampling. When the upper-layer application needs to process real-time audio variation patterns for time-domain analysis, an equal probability sampling method is used; when the upper-layer application only performs feature processing and not time-domain analysis, other random sampling methods are used. The digital sensor array includes ultrasonic array sensors, microphone arrays, infrared digital sensor arrays, or laser digital sensor arrays. The video data stream output from the image sensor is received through the DVP digital interface, and asynchronous FIFO buffering is performed on the video data stream to ensure its continuity; the video data stream is compressed using an H.264-like video data compression method. Multi-channel PCM data streams are combined and encoded, and the encoded PCM data streams are interpolated into the compressed video data stream. The encoded PCM data streams and the compressed video data streams are kept in time synchronization. The number of multi-channel PCM data frames inserted can be dynamically adjusted according to the real-time data volume. The mixed data streams are packaged and uploaded to the cloud server using gigabit network communication protocols.

2. The sensor array data transmission method according to claim 1, characterized in that: The asynchronous FIFO cache processing steps include: Upon receiving the frame synchronization signal from the image sensor, reset the input FIFO; Input data is received at the first sampling frequency and written into the input FIFO. When the amount of buffered data in the input FIFO exceeds the first preset warning threshold, the buffered data in the input FIFO is written into a storage bank in the DDR controller at the second sampling frequency. After writing a complete frame of data, a frame synchronization signal is sent to the DDR controller to perform a storage bank switching action. After receiving the frame synchronization signal, the DDR controller performs a bank switching operation during the gap between when one bank completes data reception and another bank completes data transmission. This connects the bank that has completed data reception to the output FIFO and the bank that has completed data transmission to the input FIFO, while simultaneously resetting the output FIFO. After the output FIFO is reset, cached data is read from the DDR controller's storage bank and written to it so that external data can be read at the third sampling frequency; and when the amount of written data is less than the second warning threshold, cached data is continuously read from the DDR controller's storage bank.

3. A sensor array data transmission method according to any one of claims 1 to 2, characterized in that: The specific steps for compressing video data streams using an H.264-like video data compression method include: Each frame of video image in the video data stream is binarized to obtain a grayscale image; Motion estimation is performed on the current frame video image based on the reference frame video image to obtain the motion vector and motion residual matrix; The motion residual matrix is ​​quantized, and the pixels in the current frame video image are divided into intra-frame variation, fixed amount and motion vector based on the quantized motion residual matrix. Intra-frame compression is performed on pixels in the current frame video image by sorting by difference. The compressed pixels are then encoded according to a specific encoding format to obtain the frame compressed data of the current frame video image. For variable values, pixels with consistent colors are arranged and compressed, while pixels with changing colors are categorized and compressed. For fixed values ​​and motion vectors, compression is performed by transmitting only the reference frame ID and reference position. Based on the principle of frame coding, each frame of data is encapsulated according to timestamp, reference frame ID, difference type, difference vector, and the format of frame compressed data; After performing difference sorting, the background is stripped from the change amount and / or motion vector, and the current frame video image is used as the reference frame for the next frame video image.

4. A sensor array data transmission device, characterized in that: The system includes an FPGA with a PDM digital interface and a DVP digital interface. The PDM digital interface is connected to a digital sensor array and is used to receive multi-channel PDM signal streams output by the digital sensor array. The DVP digital interface is connected to an image sensor and is used to receive video data streams. The digital sensor array includes an ultrasonic array sensor, a microphone array, an infrared digital sensor array, or a laser digital sensor array. The FPGA is also equipped with a signal processor, an image processor, a mixer, and a network data processing unit. The signal processor includes a synchronous digital modulation module, a first asynchronous buffer module, and a dynamic data segmentation and compression module. The synchronous digital modulation module is used to convert a multi-channel PDM signal stream into a multi-channel PCM data stream using synchronous digital modulation. The dynamic data segmentation and compression module is used to perform dynamic segmentation and compression processing on the multi-channel PCM data stream using a random sampling method. Specifically, when the upper-layer application needs to process real-time audio variation patterns for time-domain analysis, an equal probability sampling method is used; when the upper-layer application only performs feature processing and does not perform time-domain analysis, other random sampling methods are used. The first asynchronous buffer module is located between the synchronous digital modulation module and the dynamic data segmentation and compression module, and is used to perform asynchronous FIFO buffering processing on the multi-channel PCM data stream to achieve the continuity of the multi-channel PCM data stream. The image processor includes a video data compression module and a second asynchronous buffer module. The video data compression module is used to compress the video data stream using an H.264-like video data compression method. The second asynchronous buffer module is located before the video data compression module and is used to perform asynchronous FIFO buffering processing on the video data stream to achieve the continuity of the video data stream. The mixer is used to combine and encode multi-channel PCM data streams, and interpolate the encoded PCM data streams into the compressed video data streams using interpolation. The encoded PCM data streams and the compressed video data streams are time-synchronized. The number of multi-channel PCM data frames inserted can be dynamically adjusted according to the real-time data volume. The network data processing unit is used to package and upload the interleaved and mixed data stream to the cloud server via gigabit network communication protocol.

5. A sensor array data transmission device according to claim 4, characterized in that: The video data compression module includes: The binarization sub-block is used to binarize each frame of video image in the video data stream to obtain a grayscale image. Motion change acquisition sub-block is used to perform motion estimation on the current frame video image based on the reference frame video image, and obtain motion vectors and motion residual matrices; Intra-frame data is divided into sub-blocks to quantize the motion residual matrix, and the pixels in the current frame video image are divided into intra-frame change quantities, fixed quantities, and motion vectors based on the quantized motion residual matrix. The compression sub-block is used to perform intra-frame compression on pixels in the current frame video image by sorting by difference. The compressed pixels are then encoded according to a specific encoding format to obtain the frame compression data of the current frame video image. For variable values, pixels with consistent colors are arranged and compressed, while pixels with changing colors are categorized and compressed. For fixed values ​​and motion vectors, compression is performed by transmitting only the reference frame ID and reference position. The bitstream formatting module is used to encapsulate each frame of data according to the principles of frame encoding, including timestamp, reference frame ID, difference type, difference vector, and frame compression data format. The reference frame update module is used to strip the background from the change and / or motion vector after performing difference sorting and to use the current frame video image as the reference frame for the next frame video image.

6. A sensor array data acquisition and transmission device, characterized in that: Includes digital sensor arrays, image sensors, and sensor array data transmission devices as described in any one of claims 4 to 5; The digital sensor array uses a non-uniform oscillating element arrangement to form a multi-channel signal acquisition unit, which is used to acquire and output multi-channel PDM signal streams. The image sensor is positioned at the center of the digital sensor array and is used to acquire and output video data streams; The sensor array data transmission device processes multi-channel PDM signal streams and video image data to form a mixed data stream, which is then uploaded to the cloud server.

7. The sensor array data acquisition and transmission device according to claim 6, characterized in that: Non-uniform oscillator arrangement methods include rectangular arrays, central arrays, multi-arm spiral arrays, irregular arrays, and ring or polyhedral arrays.

Images