Sensor array data transmission method and device

Abstract

The invention provides a sensor array data transmission method which is applied to an FPGA, the FPGA is provided with a PDM digital interface and a DVP digital interface, multichannel PDM signal streams output by a digital sensor array are received through the PDM digital interface, the multichannel PDM signal streams are converted into multichannel PCM data streams in a synchronous digital modulation mode, and the multichannel PCM data streams are transmitted to the FPGA through the DVP digital interface. Dynamic fragmentation compression processing is carried out on the multi-channel PCM data flow through an equal probability sampling method; the method comprises the following steps: receiving a video data stream output by an image sensor through a DVP digital interface, and compressing the video data stream through a H264-like video data compression method; the multi-channel PCM data stream is subjected to combined coding, the coded PCM data stream is interspersed in the compressed video data stream through an interpolation method, and the coded PCM data stream and the compressed video data stream keep time synchronization; and packaging and uploading the interspersed and mixed data flow to a cloud server through a gigabit network communication protocol.

Description

Technical Field

[0001] The present invention relates to the field of data transmission, and in particular to a sensor array data transmission method and device. Background Art

[0002] In industries such as petrochemicals, electric power, and the automotive industry, inspection and maintenance of pipelines and equipment have always been an important task. Regular inspections by personnel can timely detect abnormal conditions such as gas leaks, abnormal discharges, or mechanical vibrations, which is of great significance for ensuring production and the personal and property safety of personnel.

[0003] The traditional inspection method is manual inspection. Inspectors rely on listening, observing, applying soapy water and watching bubbles to inspect and maintain equipment, pipeline leaks and other abnormalities. This is highly dangerous, has low detection efficiency and is easy to miss. Through research, it is found that whether it is gas leakage, abnormal discharge or mechanical vibration, sound will be generated, so the method of using acoustic imaging to locate the sound source is born. Acoustic imaging uses a high-sensitivity digital MEMS microphone to generate the sound pressure level distribution of sound on a plane through an array signal processing algorithm, and realizes sound visualization in the form of a color contour map. The sound distribution of the object under test is intuitively displayed in the form of photos or videos. This method can effectively make up for the shortcomings of the human ear in locating the direction of the sound source, and can bring more efficient solutions in industrial detection. For example, CN117028870A provides a natural gas leak positioning audio-visual system based on ultrasonic arrays and images.

[0004] However, the existing acoustic imaging detection accuracy, sensitivity, resolution and wide-area imaging capabilities still face huge challenges. This is because the existing ultrasonic array acquisition adopts analog acquisition method, using complex ADC conversion circuit to realize acoustic signal conversion; the unit volume is large, and it is impossible to have more channels under the same volume conditions; the accuracy consistency of multiple channels is difficult to ensure; the power consumption of the whole machine and the maintenance and production costs are increased. In addition, the existing ultrasonic array uses narrowband communication methods such as USB or serial bus communication, which is limited by the communication rate and cannot achieve a higher sampling rate; it is also impossible to continuously collect, work in real time online, and cannot achieve distributed system deployment.

[0005] In addition, with the development of 5G network technology, remote transmission and real-time analysis of ultrasound images have become possible. However, how to efficiently integrate ultrasound array technology and network communication while ensuring imaging quality and system response speed still faces technical challenges.

[0006] In order to solve the above problems, people have been seeking an ideal technical solution. Summary of the invention

[0007] The purpose of the present invention is to address the deficiencies of the prior art and thus provide a sensor array data transmission method and device and an ultrasonic array sound and image acquisition device.

[0008] In order to achieve one of the above purposes, the technical solution adopted by the present invention is: a sensor array data transmission method, applied to FPGA, the FPGA has a PDM digital interface and a DVP digital interface, the method comprising: The multi-channel PDM signal stream output by the digital sensor array is received through the PDM digital interface, the multi-channel PDM signal stream is converted into a multi-channel PCM data stream by using a synchronous digital modulation method, and the multi-channel PCM data stream is dynamically sliced ​​and compressed by a random sampling method; Receive the video data stream output by the image sensor through the DVP digital interface, and compress the video data stream through the H264-like video data compression method; Combining and encoding the multi-channel PCM data streams, and inserting the encoded PCM data streams into the compressed video data streams by interpolation, wherein the encoded PCM data streams and the compressed video data streams are synchronized in time; The interleaved and mixed data streams are packaged and uploaded to the cloud server through the Gigabit network communication protocol.

[0009] Furthermore, before the multi-channel PCM data stream is subjected to dynamic slicing and compression processing by random sampling method, the multi-channel PCM data stream is subjected to asynchronous FIFO buffering processing to achieve the continuity of the multi-channel PCM data stream; Before compressing the video data stream by using the H264-like video data compression method, the video data stream is also subjected to asynchronous FIFO buffering processing to achieve the continuity of the video data stream.

[0010] Furthermore, the step of asynchronous FIFO buffer processing includes: Receive the frame synchronization signal sent by the image sensor and reset the input FIFO; receiving input data at a first sampling frequency and writing the input data into an input FIFO, triggering the writing of the buffered data in the input FIFO into a memory bank in the DDR controller at a second sampling frequency when the amount of buffered data in the input FIFO is greater than a first preset warning threshold; after a frame of data is completely written, sending a frame synchronization signal to the DDR controller to perform a memory bank switching action; After receiving the frame synchronization signal, the DDR controller performs a bank switching action in the interval between when one bank completes data reception and another bank completes data transmission, so that the bank that completes data reception is connected to the output FIFO, and the bank that completes data transmission is connected to the input FIFO, and the output FIFO is reset at the same time; After the output FIFO is reset, cache data is read from the storage bank of the DDR controller for writing, so that the external device can read data according to the third sampling frequency; and when the amount of written data is less than the second warning threshold, cache data is continuously read from the storage bank of the DDR controller.

[0011] Furthermore, the specific steps of compressing the video data stream by using the H264-like video data compression method include: Binarize each frame of video image in the video data stream to obtain a grayscale image; Perform motion estimation on the current frame video image based on the reference frame video image to obtain a motion vector and a motion residual matrix; quantizing the motion residual matrix, and dividing the pixels in the current frame video image into intra-frame variation, fixed quantity and motion vector based on the quantized motion residual matrix; The pixels in the current frame video image are compressed intra-frame by difference sorting, and the compressed pixels are pixel-encoded according to a specific coding format to obtain frame compression data of the current frame video image; for the change amount, the pixels with the same color are arranged and compressed, and the pixels with the changed color are classified and compressed; for the fixed amount and the moving vector, only the reference frame ID and the reference position are transmitted for compression; Based on the principle of frame coding, each frame data is encapsulated according to the format of timestamp, reference frame ID, difference type, difference vector and frame compression data; After the difference is sorted, the background and the change amount and / or the motion vector are separated and the current frame video image is used as a reference frame for the next frame video image.

[0012] The second object of the present invention is to provide a sensor array data transmission device, comprising an FPGA, wherein the FPGA has a PDM digital interface and a DVP digital interface; the PDM digital interface is connected to a digital sensor array and is used to receive a multi-channel PDM signal stream output by the digital sensor array; the DVP digital interface is connected to an image sensor and is used to receive a video data stream; The FPGA is also provided with a signal processor, an image processor, a mixer and a network data processing unit; The signal processor includes a synchronous digital modulation module and a data dynamic slice compression module. The synchronous digital modulation module is used to convert a multi-channel PDM signal stream into a multi-channel PCM data stream by using a synchronous digital modulation method; the data dynamic slice compression module is used to perform dynamic slice compression processing on the multi-channel PCM data stream by a random sampling method; The image processor includes a video data compression module for compressing the video data stream by using a H264-like video data compression method; The mixer is used to combine and encode the multi-channel PCM data streams, and interleave the encoded PCM data streams into the compressed video data streams by interpolation, wherein the encoded PCM data streams and the compressed video data streams are kept in time synchronization; The network data processing unit is used to package and upload the interleaved and mixed data streams to a cloud server via a gigabit network communication protocol.

[0013] Furthermore, the signal processor further comprises a first asynchronous buffer module, which is arranged between the synchronous digital modulation module and the data dynamic slice compression module, and is used for performing asynchronous FIFO buffer processing on the multi-channel PCM data stream to achieve the continuity of the multi-channel PCM data stream; The image processing unit further comprises a second asynchronous cache module, which is arranged before the video data compression module and is used for performing asynchronous FIFO cache processing on the video data stream to achieve continuity of the video data stream.

[0014] Furthermore, the video data compression module includes: A binarization sub-block is used to perform binarization processing on each frame of video image in the video data stream to obtain a grayscale image; A motion change acquisition sub-block is used to perform motion estimation on the current frame video image based on the reference frame video image to obtain a motion vector and a motion residual matrix; The intra-frame data division sub-block is used to quantize the motion residual matrix, and divide the pixels in the current frame video image into intra-frame variation, fixed quantity and motion vector based on the quantized motion residual matrix; The compression sub-block is used to perform intra-frame compression on the pixels in the current frame video image by difference sorting, and to encode the compressed pixels according to a specific encoding format to obtain frame compression data of the current frame video image; for the change amount, the pixels with the same color are arranged and compressed, and the pixels with the changed color are classified and compressed; for the fixed amount and the motion vector, only the reference frame ID and the reference position are transmitted for compression; The code stream formatting module is used to encapsulate each frame data according to the format of timestamp, reference frame ID, difference type, difference vector and frame compression data based on the principle of frame coding; The reference frame updating module is used to separate the background from the variation and / or the motion vector after performing difference sorting and use the current frame video image as the reference frame of the next frame video image.

[0015] The third object of the present invention is to provide a sensor array data acquisition and transmission device, comprising a digital sensor array, an image sensor and the aforementioned sensor array data transmission device; The digital sensor array adopts a non-uniform arrangement of vibration elements to form a multi-channel signal acquisition unit for acquiring and outputting a multi-channel PDM signal stream; The image sensor is arranged at the center of the digital sensor array and is used to collect and output a video data stream; The sensor array data transmission device processes the multi-channel PDM signal stream and the video image data to form a mixed data stream and uploads it to the cloud server.

[0016] Furthermore, the digital sensor array includes an ultrasonic array sensor, a microphone array, an infrared digital sensor array or a laser digital sensor array.

[0017] Furthermore, the non-uniform vibrator arrangement includes a rectangular array, a central array, a multi-arm spiral array, an irregular array, a ring array or a polyhedral array.

[0018] The present invention has outstanding substantive features and significant progress compared to the prior art. Specifically, the present invention processes the multi-channel PDM signal stream output by the digital sensor array and the video data stream output by the image sensor simultaneously through an integrated data transmission method and sends them out from the same interface, thereby reducing the difficulty of post-stage processing and improving the flexibility of post-stage data processing. Based on digital design, FPGA integration and algorithm optimization, high sampling rate and more accurate signal analysis and feature extraction are achieved to eliminate external environmental interference and more accurately determine the source of abnormal sound sources; The digital sensor array is combined with synchronous digital modulation for data processing. Compared with analog processing, the sampling circuit is reduced, and the overall device BOM is more streamlined and the volume is greatly reduced. The interference of external environmental signals in the analog signal processing process is also greatly reduced, reducing the distortion of the data source and further ensuring the integrity of the data output results. It is more flexible in the post-stage digital algorithm processing. Data is uploaded through the Gigabit network communication protocol, supporting sensor signal acquisition with higher sampling rates and higher resolution image data. It has good compatibility and can be directly connected to the Internet to achieve cross-regional remote control. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flow chart of the transmission method described in Example 1 of the present invention.

[0020] Figure 2 It is a flow chart of the synchronous digital modulation method described in Example 1 of the present invention.

[0021] Figure 3 It is a schematic diagram of the H264-like video data compression method described in Example 1 of the present invention.

[0022] Figure 4 It is a schematic diagram of pixel encoding in Example 1 of the present invention.

[0023] Figure 5 It is a schematic diagram of frame coding in embodiment 1 of the present invention.

[0024] Figure 6 It is the change of bandwidth occupied by the data stream before and after the H264-like video data is compressed in Example 1 of the present invention.

[0025] Figure 7 It is the interpolation reference timing in Embodiment 1 of the present invention.

[0026] Figure 8 It is a flow chart of the transmission method described in Example 2 of the present invention.

[0027] Fig. 9 It is a schematic diagram of the asynchronous FIFO cache processing described in Example 2 of the present invention.

[0028] Fig.10 It is a schematic diagram of the structure of the sensor array data transmission device described in Example 3 of the present invention.

[0029] Fig.11 It is a structural schematic diagram of the sensor array data acquisition and transmission device described in Example 4 of the present invention.

[0030] Fig.12 This is the method of the sensor array data acquisition and transmission device described in Example 4 of the present invention.

[0031] In the figure: 1. Digital sensor array; 2. Image sensor. DETAILED DESCRIPTION

[0032] The technical solution of the present invention is further described in detail below through specific implementation methods.

[0033] Example 1 This embodiment provides a sensor array data transmission method, which is applied to an FPGA, wherein the FPGA has a PDM digital interface and a DVP digital interface, such as Figure 1As shown, the method includes: The multi-channel PDM signal stream output by the digital sensor array 1 is received through the PDM digital interface, the multi-channel PDM signal stream is converted into a multi-channel PCM data stream by adopting a synchronous digital modulation method, and the multi-channel PCM data stream is dynamically sliced ​​and compressed by a random sampling method.

[0034] The digital sensor array 1 outputs a PDM carrier signal. The PDM signal is a modulation method that uses a digital signal to represent an analog signal, that is, a frequency carrier signal. The advantage of PDM is that it can highly restore the real analog signal and reduce signal distortion. However, PDM is essentially an analog signal and cannot be directly recognized by the upper-level algorithm, so the PDM signal needs to be converted into a digital PCM signal.

[0035] The synchronous digital modulation algorithm used in this embodiment converts the PDM signal into a digital PCM signal. Specifically, the synchronous digital modulation algorithm is implemented based on FPGA and restores the analog signal through multiple integration and sampling. Specifically, Figure 2 As shown, it includes a 1st order integrator, a 2nd order integrator, a decimation / down sampling module, a 1st order comb filter and a 2nd order comb filter.

[0036] Specifically, random sampling methods include simple random sampling, equal probability sampling, stratified sampling, and cluster sampling. The specific selection can be based on the specific application. For example, when the upper-layer application needs to process the real-time audio change law for time domain analysis, the equal probability sampling method needs to be used; if the upper-layer application only performs feature processing and does not perform time domain analysis, other random sampling methods can be used.

[0037] The video data stream output by the image sensor 2 is received via the DVP digital interface, and the video data stream is compressed using a H264-like video data compression method. Preferably, the image sensor 2 is a camera.

[0038] In one embodiment, if Figure 3 As shown, the specific steps of compressing the video data stream by using the H264-like video data compression method include: Each frame of video image in the video data stream is binarized to obtain a grayscale image; the binarized image has a more uniform color depth and is more conducive to subsequent data processing.

[0039] Motion estimation is performed on the current frame video image based on the reference frame video image to obtain a motion vector and a motion residual matrix. Motion estimation is the process of finding the optimal or suboptimal motion vector when the current frame image undergoes motion changes relative to the reference frame image. The motion change means that the color of a certain pixel point between the current frame image and the reference frame image has changed. Specifically, the motion change is divided into three categories: local change, overall movement, and full dynamic. However, since the application scenarios of this embodiment are mostly fixed monitoring, only two types of local changes and overall movement need to be considered.

[0040] It should be noted that the reference frame changes in real time as the frame moves. Generally, the previous video frame is used as the reference frame for the next video frame. The size of the motion vector affects the data compression rate. If the variable is small, the compression rate is high, otherwise the compression rate is low.

[0041] quantizing the motion residual matrix, and dividing the pixels in the current frame video image into intra-frame variation, fixed quantity and motion vector based on the quantized motion residual matrix; The pixels in the current frame video image are compressed intra-frame by difference sorting, and the compressed pixels are pixel-encoded according to a specific encoding format to obtain the frame compression data of the current frame video image; for the change amount, the pixels with the same color are arranged and compressed, and the pixels with the changed color are classified and compressed; for the fixed amount and the moving vector, only the reference frame ID and the reference position are transmitted for compression; Figure 4 An embodiment of intra-frame compression and pixel encoding is shown.

[0042] It is understandable that encoding corresponds to decoding, so the basic requirement of encoding is to facilitate decoding in order to improve efficiency.

[0043] Based on the principle of frame coding, each frame data is encapsulated according to the format of timestamp, reference frame ID, difference type, difference vector and frame compression data. Figure 5 shown.

[0044] like Figure 6 As shown, the actual comparison shows the change in bandwidth occupied by the data stream before and after data compression.

[0045] It can be seen that the actual image effect is within an acceptable range, the frame rate is controlled at 15fps, and the size of each frame image is about 14KByte.

[0046] Furthermore, compared with the standard H264 inter-frame prediction compression algorithm, the standard H264 inter-frame prediction compression algorithm predicts and judges the original color of the image, can highly restore the real image changes, and is excellent in image distortion and color restoration; but the disadvantage is that it has high requirements for computing power and hardware. The advantages and disadvantages of the H264-like video data compression method in this embodiment are also very obvious. Due to the use of grayscale quantization, color accuracy is lost, and compression is limited to the inter-frame motion difference; Advantages: greatly shorten the data volume of the image stream data, Disadvantages: poor compression effect on fast-moving targets and poor image restoration; However, since this embodiment is directional monitoring and fixed-position monitoring, it does not affect the compression effect.

[0047] It can be understood that after the difference ordering is performed, the background and the variation and / or the motion vector are separated and the current frame video image is used as a reference frame for the next frame video image.

[0048] After the multi-channel PCM data stream and the video data stream are compressed, the multi-channel PCM data stream is combined and encoded because the PDM signal is multi-channel. Further, after the multi-channel PCM data stream is combined and encoded, the encoded PCM data stream is interpolated into the compressed video data stream by interpolation, wherein the encoded PCM data stream and the compressed video data stream are synchronized in time.

[0049] Under normal circumstances, multi-channel PCM data streams occupy less bandwidth, while video data streams occupy more bandwidth. Therefore, under the premise that both share the network bandwidth for transmission, in order to synchronize video and audio, it is necessary to insert the multi-channel PCM data stream between the video data streams, and at the same time maintain the time synchronization relationship between the multi-channel PCM data stream and the video data stream; For specific operations, refer to Figure 7 Specifically, the number of audio frames to be inserted can be dynamically adjusted according to the amount of real-time data.

[0050] Finally, the interleaved and mixed data streams are packaged and uploaded to the cloud server through the Gigabit network communication protocol.

[0051] It should be noted that before uploading the interleaved and mixed data streams to the cloud server through the Gigabit network communication protocol, it is necessary to configure the IP and server addresses on the device and establish an ARP connection with the server; when ready, wait to receive the interleaved and mixed data streams.

[0052] It can be seen that this embodiment uses an integrated data transmission method to simultaneously process the multi-channel PDM signal stream output by the digital sensor array 1 and the video data stream output by the image sensor 2 and send them out from the same interface, thereby reducing the difficulty of post-stage processing and improving the flexibility of post-stage data processing; Based on digital design, FPGA integration and algorithm optimization, high sampling rate and more accurate signal analysis and feature extraction are achieved to eliminate external environmental interference and more accurately determine the source of abnormal sound.

[0053] Data is uploaded through the Gigabit network communication protocol, supporting sensor signal acquisition with higher sampling rates and higher resolution image data. It has good compatibility and can be directly connected to the Internet to achieve cross-regional remote control.

[0054] Example 2 This embodiment provides another implementation of the sensor array data transmission method.

[0055] In this embodiment, if Figure 8 As shown, before the multi-channel PCM data stream is subjected to dynamic slicing compression processing by random sampling method, the multi-channel PCM data stream is also subjected to asynchronous FIFO buffer processing to achieve the continuity of the multi-channel PCM data stream; Before compressing the video data stream by using the H264-like video data compression method, the video data stream is also subjected to asynchronous FIFO buffering processing to achieve the continuity of the video data stream.

[0056] Specifically, asynchronous FIFO buffering is performed on multi-channel PCM data streams to achieve 1024-bit input and 32-bit output; asynchronous FIFO buffering is performed on video data streams to achieve 32-bit input and 256-bit output.

[0057] The steps of the asynchronous FIFO buffer processing are the same. For ease of understanding, the asynchronous FIFO buffer processing of the video data stream is taken as an example. Fig. 9 , details the steps of the asynchronous FIFO buffer processing, the specific steps include: Receiving the frame synchronization signal sent by the image sensor 2, resetting the input FIFO; Receive input data according to a first sampling frequency (for example, the sampling clock is 50M), and write the input data into an input FIFO. When the amount of cached data in the input FIFO is greater than a first preset warning threshold, trigger the cached data in the input FIFO to be written into a memory bank in the DDR controller at a second sampling frequency (for example, the sampling clock is 100M); after a frame of data is completely written, send a frame synchronization signal to the DDR controller to perform a memory bank switching action; generally, the first preset warning threshold is not greater than 1 / 2 of the rated cache amount of the input FIFO; After receiving the frame synchronization signal, the DDR controller switches the storage bank in the interval between when one storage bank completes data reception and the other storage bank completes data transmission, so that the storage bank that completes data reception is connected to the output FIFO, and the storage bank that completes data transmission is connected to the input FIFO, and the output FIFO is reset at the same time; it can be understood that the DDR controller has at least two storage banks, which can meet the scheduling within one cycle of frame transmission and reception; the purpose of setting two storage banks is to ensure the integrity of the frame during the data flow conversion process at different frequencies, and to ensure the flow characteristics of the frame data. It should be noted that the current input frame and output frame cannot be the same storage bank.

[0058] After the output FIFO is reset, the cache data is read from the memory bank of the DDR controller for writing, so that the external device can read the data according to the third sampling frequency (for example, the sampling clock is 125M); and when the amount of written data is less than the second warning threshold, the cache data is continuously read from the memory bank of the DDR controller. The second warning threshold is not less than 1 / 2 of the rated cache amount of the output FIFO.

[0059] It should be noted that within the valid frame interval, it is necessary to ensure that there is data in the output FIFO at all times; when there is data in the output FIFO, the sending module needs to schedule the sending mechanism as soon as possible to send the data out.

[0060] The above asynchronous buffering process is implemented by input and output FIFO and FPGA internal BRAM, and each buffer is 2K deep; and the frame buffer needs to meet the needs of a complete frame, so it is implemented using external DDR.

[0061] Example 3 This embodiment provides a sensor array data transmission device, such as Fig.10 As shown, it includes an FPGA, and the FPGA has a PDM digital interface and a DVP digital interface; the PDM digital interface is connected to the digital sensor array 1, and is used to receive the multi-channel PDM signal stream output by the digital sensor array 1; the DVP digital interface is connected to the image sensor 2, and is used to receive the video data stream; The FPGA is also provided with a signal processor, an image processor, a mixer and a network data processing unit; The signal processor includes a synchronous digital modulation module and a data dynamic slice compression module. The synchronous digital modulation module is used to convert a multi-channel PDM signal stream into a multi-channel PCM data stream by using a synchronous digital modulation method; the data dynamic slice compression module is used to perform dynamic slice compression processing on the multi-channel PCM data stream by a random sampling method; The image processor includes a video data compression module for compressing the video data stream by using a H264-like video data compression method; The mixer is used to combine and encode the multi-channel PCM data streams, and interleave the encoded PCM data streams into the compressed video data streams by interpolation, wherein the encoded PCM data streams and the compressed video data streams are kept in time synchronization; The network data processing unit is used to package and upload the interleaved and mixed data streams to a cloud server via a gigabit network communication protocol.

[0062] Furthermore, the signal processor further comprises a first asynchronous buffer module, which is arranged between the synchronous digital modulation module and the data dynamic slice compression module, and is used for performing asynchronous FIFO buffer processing on the multi-channel PCM data stream to achieve the continuity of the multi-channel PCM data stream; The image processing unit further comprises a second asynchronous cache module, which is arranged before the video data compression module and is used for performing asynchronous FIFO cache processing on the video data stream to achieve continuity of the video data stream.

[0063] Specifically, Fig. 9 As shown, the steps of the asynchronous FIFO buffer processing include: Receiving the frame synchronization signal sent by the image sensor 2, resetting the input FIFO; Receive input data according to a first sampling frequency (for example, the sampling clock is 50M), and write the input data into an input FIFO. When the amount of cached data in the input FIFO is greater than a first preset warning threshold, trigger the cached data in the input FIFO to be written into a memory bank in the DDR controller at a second sampling frequency (for example, the sampling clock is 100M); after a frame of data is completely written, send a frame synchronization signal to the DDR controller to perform a memory bank switching action; generally, the first preset warning threshold is not greater than 1 / 2 of the rated cache amount of the input FIFO; After receiving the frame synchronization signal, the DDR controller switches the storage bank in the interval between when one storage bank completes data reception and the other storage bank completes data transmission, so that the storage bank that completes data reception is connected to the output FIFO, and the storage bank that completes data transmission is connected to the input FIFO, and the output FIFO is reset at the same time; it can be understood that the DDR controller has at least two storage banks, which can meet the scheduling within one cycle of frame transmission and reception; the purpose of setting two storage banks is to ensure the integrity of the frame during the data flow conversion process at different frequencies, and to ensure the flow characteristics of the frame data. It should be noted that the current input frame and output frame cannot be the same storage bank.

[0064] After the output FIFO is reset, the cache data is read from the memory bank of the DDR controller for writing, so that the external device can read the data according to the third sampling frequency (for example, the sampling clock is 125M); and when the amount of written data is less than the second warning threshold, the cache data is continuously read from the memory bank of the DDR controller. The second warning threshold is not less than 1 / 2 of the rated cache amount of the output FIFO.

[0065] It should be noted that within the valid frame interval, it is necessary to ensure that there is data in the output FIFO at all times; when there is data in the output FIFO, the sending module needs to schedule the sending mechanism as soon as possible to send the data out.

[0066] Furthermore, the video data compression module includes: A binarization sub-block is used to perform binarization processing on each frame of video image in the video data stream to obtain a grayscale image; A motion change acquisition sub-block is used to perform motion estimation on the current frame video image based on the reference frame video image to obtain a motion vector and a motion residual matrix; The intra-frame data division sub-block is used to quantize the motion residual matrix, and divide the pixels in the current frame video image into intra-frame variation, fixed quantity and motion vector based on the quantized motion residual matrix; The compression sub-block is used to perform intra-frame compression on the pixels in the current frame video image by difference sorting, and to encode the compressed pixels according to a specific encoding format to obtain frame compression data of the current frame video image; for the change amount, the pixels with the same color are arranged and compressed, and the pixels with the changed color are classified and compressed; for the fixed amount and the motion vector, only the reference frame ID and the reference position are transmitted for compression; The code stream formatting module is used to encapsulate each frame data according to the format of timestamp, reference frame ID, difference type, difference vector and frame compression data based on the principle of frame coding; The reference frame updating module is used to separate the background from the variation and / or the motion vector after performing difference sorting and use the current frame video image as the reference frame of the next frame video image.

[0067] Example 4 This embodiment provides a sensor array data acquisition and transmission device, such as Fig.11 As shown, it specifically includes a digital sensor array 1, an image sensor 2 and the sensor array data transmission device described in Example 3. It can be understood that it also includes a power supply unit, which integrates multiple power supplies to adapt to the stable and reliable operation of the digital sensor array 1, processing unit, image acquisition, data transmission and other functions; and provides necessary exception handling mechanisms, which can effectively and timely cut off the power supply in the event of overheating, overload and other conditions to ensure the safety of the equipment.

[0068] The digital sensor array 1 includes an ultrasonic array sensor, a microphone array, an infrared digital sensor array 1 or a laser digital sensor array 1. Specifically, no matter which type of digital sensor array 1 is used, it outputs a sensor signal in a PDM format.

[0069] This embodiment takes an ultrasonic array sensor as an example, which includes a plurality of high-sensitivity ultrasonic sensors, which are generally microphones; the plurality of high-sensitivity ultrasonic sensors are arranged in a specific array to form an ultrasonic data acquisition unit with multiple channels to collect multi-channel data. These data are processed by a specific algorithm, such as DOA (beamforming algorithm), to accurately obtain feature data with directional attributes, which can be used for sound source positioning.

[0070] Furthermore, the non-uniform element arrangement includes a rectangular array, a central array, a multi-arm spiral array, an irregular array, a ring array or a polyhedral array. In practice, the array layout is not limited to the above, and different arrangements can be suitable for different application scenarios.

[0071] The multi-channel array element layout can obtain data acquisition capabilities of at least 16 channels and up to 512 channels, which can increase the acquisition accuracy and realize wide-angle and multi-phase data acquisition. In practice, there is no limit on the number of multiple channels, and 64 channels are usually more common. Different application scenarios have different requirements for data accuracy. In general, by increasing the number of channels, the number of samples can be increased, and the accuracy of the output results can be improved.

[0072] The image sensor 2 is integrated with the digital sensor array 1. Specifically, the image sensor 2 is located at the center of the digital sensor array 1, so that the image sensor 2 and the digital sensor array 1 are located in the same vector direction, which facilitates the organic fusion of different types of data.

[0073] In a specific implementation, the image sensor 2 uses a high-definition camera to achieve high-definition image acquisition and high-definition video recording, thereby improving the clarity of sound and image while reducing artifacts, ghosting and other phenomena.

[0074] The sensor array data transmission device comprises an FPGA, wherein the FPGA has a PDM digital interface and a DVP digital interface; the PDM digital interface is connected to the digital sensor array 1 and is used to receive a multi-channel PDM signal stream output by the digital sensor array 1; the DVP digital interface is connected to the image sensor 2 and is used to receive a video data stream; The FPGA is also provided with a signal processor, an image processor, a mixer and a network data processing unit; The signal processor includes a synchronous digital modulation module and a data dynamic slice compression module. The synchronous digital modulation module is used to convert a multi-channel PDM signal stream into a multi-channel PCM data stream by using a synchronous digital modulation method; the data dynamic slice compression module is used to perform dynamic slice compression processing on the multi-channel PCM data stream by a random sampling method; The image processor includes a video data compression module for compressing the video data stream by using a H264-like video data compression method; The mixer is used to combine and encode the multi-channel PCM data streams, and interleave the encoded PCM data streams into the compressed video data streams by interpolation, wherein the encoded PCM data streams and the compressed video data streams are kept in time synchronization; The network data processing unit is used to package and upload the interleaved and mixed data streams to a cloud server via a gigabit network communication protocol.

[0075] The network data processing unit is used to package and send the ultrasound video stream.

[0076] Specifically, Fig.12 As shown, the working principle of the ultrasonic array sensor is as follows: The ultrasonic array detector collects ultrasonic raw digital data of multiple channels; the sensor processing unit performs logic processing and filtering processing on the multi-channel ultrasonic data to restore the ultrasonic data stream; The image sensor 2 collects image data; the image processing unit performs logic processing and noise reduction processing on the image data to restore high-definition image data.

[0077] The high-speed data fusion unit is used to merge the ultrasound data stream and the high-definition image data to form an ultrasound video stream, which is packaged and sent out by the network data processing unit.

[0078] After obtaining the ultrasound data stream and the video data stream, asynchronous FIFO buffering is performed to ensure the continuity of the data stream. Specifically, Figure 8 As shown, asynchronous FIFO buffering is performed on multi-channel PCM data streams to achieve 1024-bit input and 32-bit output; asynchronous FIFO buffering is performed on video data streams to achieve 32-bit input and 256-bit output.

[0079] Among them, before packaging the interspersed and mixed data streams and uploading them to the cloud server through the Gigabit network communication protocol, it is necessary to first configure the IP and server addresses on the device, and establish an ARP connection with the server; when ready, wait to receive the interspersed and mixed data streams.

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or some technical features can be replaced by equivalents without departing from the spirit of the technical solution of the present invention, which should be included in the scope of the technical solution for protection of the present invention.

Claims

1. A sensor array data transmission method, applied to an FPGA, wherein the FPGA has a PDM digital interface and a DVP digital interface, characterized in that: The multi-channel PDM signal stream output by the digital sensor array is received through the PDM digital interface, the multi-channel PDM signal stream is converted into a multi-channel PCM data stream by using a synchronous digital modulation method, and the multi-channel PCM data stream is dynamically sliced ​​and compressed by a random sampling method; The video data stream output by the image sensor is received through the DVP digital interface, and the video data stream is compressed using a H264-like video data compression method; Combining and encoding the multi-channel PCM data streams, and inserting the encoded PCM data streams into the compressed video data streams by interpolation, wherein the encoded PCM data streams and the compressed video data streams are synchronized in time; The interleaved and mixed data streams are packaged and uploaded to the cloud server through the Gigabit network communication protocol.

2. A sensor array data transmission method according to claim 1, characterized in that: Before dynamically slicing and compressing the multi-channel PCM data stream by random sampling, asynchronous FIFO buffering is performed on the multi-channel PCM data stream to achieve the continuity of the multi-channel PCM data stream; Before compressing the video data stream by using the H264-like video data compression method, the video data stream is also subjected to asynchronous FIFO buffering processing to achieve the continuity of the video data stream.

3. A sensor array data transmission method according to claim 2, characterized in that: The steps of the asynchronous FIFO buffer processing include: Receive the frame synchronization signal sent by the image sensor and reset the input FIFO; receiving input data at a first sampling frequency and writing the input data into an input FIFO, triggering the writing of the buffered data in the input FIFO into a memory bank in the DDR controller at a second sampling frequency when the amount of buffered data in the input FIFO is greater than a first preset warning threshold; after a frame of data is completely written, sending a frame synchronization signal to the DDR controller to perform a memory bank switching action; After receiving the frame synchronization signal, the DDR controller performs a bank switching action in the interval between when one bank completes data reception and another bank completes data transmission, so that the bank that completes data reception is connected to the output FIFO, and the bank that completes data transmission is connected to the input FIFO, and the output FIFO is reset at the same time; After the output FIFO is reset, cache data is read from the storage bank of the DDR controller for writing, so that the external device can read data according to the third sampling frequency; and when the amount of written data is less than the second warning threshold, cache data is continuously read from the storage bank of the DDR controller.

4. A sensor array data transmission method according to any one of claims 1 to 3, characterized in that: The specific steps of compressing the video data stream by the H264-like video data compression method include: Binarize each frame of video image in the video data stream to obtain a grayscale image; Perform motion estimation on the current frame video image based on the reference frame video image to obtain a motion vector and a motion residual matrix; quantizing the motion residual matrix, and dividing the pixels in the current frame video image into intra-frame variation, fixed quantity and motion vector based on the quantized motion residual matrix; The pixels in the current frame video image are compressed intra-frame by difference sorting, and the compressed pixels are pixel-encoded according to a specific coding format to obtain frame compression data of the current frame video image; for the change amount, the pixels with the same color are arranged and compressed, and the pixels with the changed color are classified and compressed; for the fixed amount and the moving vector, only the reference frame ID and the reference position are transmitted for compression; Based on the principle of frame coding, each frame data is encapsulated according to the format of timestamp, reference frame ID, difference type, difference vector and frame compression data; After the difference is sorted, the background and the variation and / or the motion vector are separated and the current frame of the video image is used as a reference frame for the next frame of the video image.

5. A sensor array data transmission device, characterized in that: The device comprises an FPGA, wherein the FPGA has a PDM digital interface and a DVP digital interface; the PDM digital interface is connected to a digital sensor array and is used to receive a multi-channel PDM signal stream output by the digital sensor array; the DVP digital interface is connected to an image sensor and is used to receive a video data stream; The FPGA is also provided with a signal processor, an image processor, a mixer and a network data processing unit; The signal processor includes a synchronous digital modulation module and a data dynamic slice compression module. The synchronous digital modulation module is used to convert a multi-channel PDM signal stream into a multi-channel PCM data stream by using a synchronous digital modulation method; the data dynamic slice compression module is used to perform dynamic slice compression processing on the multi-channel PCM data stream by a random sampling method; The image processor includes a video data compression module for compressing the video data stream by using a H264-like video data compression method; The mixer is used to combine and encode the multi-channel PCM data streams, and interleave the encoded PCM data streams into the compressed video data streams by interpolation, wherein the encoded PCM data streams and the compressed video data streams are kept in time synchronization; The network data processing unit is used to package and upload the interleaved and mixed data streams to a cloud server via a gigabit network communication protocol.

6. A sensor array data transmission device according to claim 5, characterized in that: The signal processor further comprises a first asynchronous buffer module, which is arranged between the synchronous digital modulation module and the data dynamic slice compression module and is used for performing asynchronous FIFO buffer processing on the multi-channel PCM data stream to achieve the continuity of the multi-channel PCM data stream; The image processing unit further comprises a second asynchronous cache module, which is arranged before the video data compression module and is used for performing asynchronous FIFO cache processing on the video data stream to achieve continuity of the video data stream.

7. A sensor array data transmission device according to claim 5, characterized in that: The video data compression module comprises: A binarization sub-block is used to perform binarization processing on each frame of video image in the video data stream to obtain a grayscale image; A motion change acquisition sub-block is used to perform motion estimation on the current frame video image based on the reference frame video image to obtain a motion vector and a motion residual matrix; The intra-frame data division sub-block is used to quantize the motion residual matrix, and divide the pixels in the current frame video image into intra-frame variation, fixed quantity and motion vector based on the quantized motion residual matrix; The compression sub-block is used to perform intra-frame compression on the pixels in the current frame video image by difference sorting, and to encode the compressed pixels according to a specific encoding format to obtain frame compression data of the current frame video image; for the change amount, the pixels with the same color are arranged and compressed, and the pixels with the changed color are classified and compressed; for the fixed amount and the motion vector, only the reference frame ID and the reference position are transmitted for compression; The code stream formatting module is used to encapsulate each frame data according to the format of timestamp, reference frame ID, difference type, difference vector and frame compression data based on the principle of frame coding; The reference frame updating module is used to separate the background from the variation and / or the motion vector after performing difference sorting and use the current frame video image as the reference frame of the next frame video image.

8. A sensor array data acquisition and transmission device, characterized in that: It comprises a digital sensor array, an image sensor and a sensor array data transmission device according to any one of claims 5 to 7; The digital sensor array adopts a non-uniform arrangement of vibration elements to form a multi-channel signal acquisition unit for acquiring and outputting a multi-channel PDM signal stream; The image sensor is arranged at the center of the digital sensor array and is used to collect and output a video data stream; The sensor array data transmission device processes the multi-channel PDM signal stream and the video image data to form a mixed data stream and uploads it to the cloud server.

9. A sensor array data acquisition and transmission device according to claim 8, characterized in that: The digital sensor array includes an ultrasonic array sensor, a microphone array, an infrared digital sensor array or a laser digital sensor array.

10. The sensor array data acquisition and transmission device according to claim 8, characterized in that: The non-uniform element arrangements include rectangular arrays, central arrays, multi-arm spiral arrays, irregular arrays, ring arrays or polyhedral arrays.

Images