Configuration-based correlation acquisition method and computing device based on underwater acoustic data

Abstract

Provided are a configuration-based, correlated acquisition and transmission control method and communication device for underwater acoustic data. The provided data correlation acquisition and transmission control method includes: a device message information acquisition module that uses a loaded data parsing dynamic library to generate a device message data packet, appends current local timing information to the generated device message data packet, and stores the packet in a buffer; an audio data acquisition module that acquires one or more audio data from a sound card, audio acquisition card, and / or audio acquisition device, appends current local timing information to the generated audio data packet, and stores the packet in a buffer; and a data packaging and transmission module that retrieves device message data packets, audio data packets, self-noise data packets, and / or array element domain data packets from the buffer at one or more time intervals and transmits the data via the network.

Description

Technical Field

[0001] The present application belongs to the field of data acquisition and transmission control, and specifically relates to a configuration-related acquisition and transmission control method, and its communication equipment and system, which can automatically acquire, parse and process various configuration data, classify data with high performance, and transmit underwater acoustic data through a network. Background Art

[0002] During a ship's voyage, the operation of its equipment generates a large amount of data, such as sonar (audio) data, scene (video) data, service (log) data, and array element data. This data is diverse, with inconsistent formats and interfaces, inconsistent data generation and usage cycles, and inconsistent data generation and collection methods.

[0003] Traditional data recording methods rely on users to collect separate data types. For example, target noise data is collected using a recording device, scene data is collected using a handheld camera, and service data is recorded using pen and paper. Other data types are also collected and recorded separately using different methods. When users record data, there is no guarantee that the data is collected at the same time, and the frequency of data collection varies, making it difficult to correlate data. This manual data collection method is inefficient and labor-intensive. Furthermore, inevitable manual errors can cause the collected data to deviate from the actual situation. Furthermore, this manual data collection method makes subsequent statistics on large data volumes extremely difficult. Manually recorded data cannot easily reconstruct the scene at the time of data collection, making further analysis and research difficult. Furthermore, offline data transfer is not conducive to data utilization.

[0004] Therefore, it is hoped to provide a method and communication equipment for the associated collection and real-time transmission of multiple data with different sources, different transmission modes and different periods, and to realize the associated collection and transmission of ship configuration data by controlling the process of data collection and transmission. Summary of the Invention

[0005] To address one or more issues existing in the prior art, embodiments of the present application provide a configurable, correlated acquisition and transmission control method based on underwater acoustic data. This method automatically collects and processes various types of data acquired onboard ships. The collected data is then synchronized and correlated with other data, including array domain data, self-noise data, scene data, and underwater acoustic environment data, with underwater acoustic service data as the primary focus and sonar target data as the core. This allows the diverse and large amounts of data generated onboard ships to be transformed into data products or services that are easily analyzed, processed, and utilized.

[0006] To solve the problem of collecting and integrating various types of discrete data into structured data so that various types of data form an association relationship. The collection of data occurs at a specific time. For example, sonar data forms a continuous data stream, in which the data of a specific target appears at a specific time point or time period. Accordingly, the self-noise data at that time point is needed to assist in the identification of the target. However, during data collection, the moment when the target appears is not known, and the target can only be detected after the subsequent analysis and processing of the audio data. Therefore, during data collection, it is also necessary to establish an association between various data from different devices and different data sources, for example, through time. According to an embodiment of the present application, a distributed collection method for various types of data is adopted, and the associated collection of configuration data is realized by integrating and associating various types of data, and multiple tasks for collecting various types of data are uniformly scheduled. The configured associated collection method and system based on underwater acoustic data according to the embodiment of the present application are realized in a distributed and modular manner, which facilitates the improvement and expansion of subsequent data collection functions, and also facilitates the support of newly added data types to be collected and devices for collecting data.

[0007] Among them, sonar target data refers to the sonar target audio data and its annotation information obtained from the beam domain, as well as the modulation spectrum data, low-frequency line spectrum data, time-frequency data, feature data, etc. obtained by synchronous processing of the sonar target audio data; underwater acoustic service data refers to the data generated by the sonar unit during sonar search, discovery, matching analysis, tracking of targets, and support platform operations.

[0008] According to the first aspect of the present application, a first data association collection and transmission control method according to the first aspect of the present application is provided, including: obtaining time from a timing device at a specified time interval and updating local time information with the obtained time; obtaining sonar target tracking information at a specified time interval; the device message information collection module loads one or more data parsing dynamic libraries corresponding to the device information collection unit, creates threads corresponding to the data types according to one or more data types provided by the device information collection unit, adds a multicast group corresponding to the data type to each created thread, and uses the first data parsing dynamic library corresponding to the data type of the loaded one or more data parsing dynamic libraries to collect data. The state library receives and parses the multicast data packet according to its own data parsing method to obtain the first data provided by the device information acquisition unit, accumulates one or more copies of the first data to generate a device message data packet, appends the current local time information to the generated device message data packet and stores it in a buffer; the audio data acquisition module collects one or more audio data from the sound card, audio acquisition card and / or audio acquisition device, accumulates one or more copies of the same type of audio data to generate an audio data packet, appends the current local time information to the generated audio data packet and stores it in a buffer; the self-noise data acquisition module creates a self-noise data acquisition thread, and in the created self-noise data acquisition thread, it generates an audio data packet based on the specified network address. The self-noise sampling data packet sent by the noise monitoring system is received at an interval, the sampling of the self-noise data is obtained from the self-noise sampling data packet, and the sampling of the self-noise data is converted to obtain the self-noise data; the self-noise data is accumulated to generate a self-noise data packet; the current local time information and the current sonar target tracking information are attached to the generated self-noise data packet and stored in the buffer; the array element domain data acquisition and parsing submodule creates an array element domain data acquisition thread, and in the array element domain data acquisition thread, cyclically receives the array element domain data subpackets sent by the array element domain data preprocessing submodule from the specified multicast address, accumulates the array element domain data from the array element domain data subpackets to generate an array element domain data packet; the current local time information and the current sonar target tracking information are attached to the generated self-noise data packet and stored in the buffer; the array element domain data acquisition and parsing submodule creates an array element domain data acquisition thread, and cyclically receives the array element domain data subpackets sent by the array element domain data preprocessing submodule from the specified multicast address in the array element domain data acquisition thread, and accumulates the array element domain data from the array element domain data subpackets to generate an array element domain data packet; The system information and the current sonar target tracking information are appended to the generated array element domain data packet and stored in a buffer; the data packaging and sending module respectively obtains the device message data packet, the audio data packet, the self-noise data packet and / or the array element domain data packet from the buffer at one or more time intervals, and sends them through the network; wherein the timer thread of the exclusive CPU core counts the clock of the CPU core to obtain a microsecond clock, and the one or more time intervals are timed by the microsecond clock, so as to trigger the data packaging and sending module to obtain one or more of the device message data packet, the audio data packet, the self-noise data packet and the array element domain data packet in response to the expiration of the one or more time intervals, and send them through the network.

[0009] According to the first data association acquisition and transmission control method of the first aspect of the present application, a second data association acquisition and transmission control method according to the first aspect of the present application is provided, which also includes: a scene data acquisition module creates a corresponding thread for each video to be collected, obtains an audio data stream and a video data stream based on the RTSP protocol in the created thread, decodes the obtained audio data stream and video data stream respectively to obtain audio stream data and video stream data, encodes the audio stream data and the video stream data into a specified format respectively, merges the encoded audio stream data and video stream data and generates a scene data packet; appends the current local time information and the current sonar target tracking information to the generated scene data packet and stores it in a buffer; a data packaging and sending module obtains the scene data packet from the buffer at a specified time interval and sends it through the network; wherein the data packaging and sending module is triggered to obtain the scene data packet and send it through the network in response to the expiration of the specified time interval of the microsecond clock timing.

[0010] According to the second data association acquisition and transmission control method of the first aspect of the present application, a third data association acquisition and transmission control method according to the first aspect of the present application is provided, wherein before acquiring scene data, the scene acquisition module sends an acquisition start instruction to acquire the scene data; for multiple videos existing at the same time, multiple corresponding threads are created to acquire each video separately; the video stream address is obtained through the avformat_open_input() function provided by the FFmpeg codec library, and the video stream information is obtained through the avformat_find_stream_info() function to obtain the AVFormatContext structure; by parsing the nb_streams member variable in the AVFormatContext structure, the number of audio data streams and video data streams is obtained, and the encoding type of the data stream is judged according to the enumeration value AVMediaType; for the encoded audio stream data and video stream data, the av_read_frame() function is used to cyclically acquire the data stream data subpacket data, and the subpacket data in the format of the AVPacket structure is returned; the audio stream data and the video stream data are merged using the av_interleaved_write_frame() function.

[0011] According to one of the first to third data association acquisition and transmission control methods of the first aspect of the present application, a fourth data association acquisition and transmission control method according to the first aspect of the present application is provided, wherein the self-noise data acquisition module samples the self-noise data using the formula

[0012]

[0013] The self-noise data is obtained by conversion, where a is the sensor sensitivity, b is the conditioning gain of the conditioning board of the noise monitoring system, c is the A / D gain of the AD board of the noise monitoring system, and x is the sampling of the self-noise data.

[0014] According to the fourth data association acquisition and transmission control method of the first aspect of the present application, a fifth data association acquisition and transmission control method according to the first aspect of the present application is provided, wherein for the sound card, the audio data acquisition module monitors the sound card input device to collect audio data from the WaveInEvent event; for the audio acquisition card, the audio data acquisition module captures the data packet provided by the audio acquisition card from the network adapter and parses the audio data according to the audio sampling rate and data length; for the audio acquisition device, the audio data acquisition module receives audio data from the multicast address corresponding to the audio acquisition device, and also queries the status of the audio acquisition device through UDP multicast commands, and sends audio start acquisition and stop acquisition instructions to control the audio acquisition device to send or stop sending audio data.

[0015] According to the fifth data association collection and transmission control method of the first aspect of the present application, a sixth data association collection and transmission control method according to the first aspect of the present application is provided, wherein the microsecond timer of the timer thread times a 40ms time interval to trigger the collection and transmission of audio data packets from the buffer area.

[0016] According to the sixth data association collection and transmission control method of the first aspect of the present application, a seventh data association collection and transmission control method according to the first aspect of the present application is provided, wherein in response to the use of a new device information collection unit, the device message information collection module loads a new data parsing dynamic library corresponding to the data type provided by the new device information collection unit; the loaded data parsing dynamic library uses a delegation method to receive and parse multicast data packets, and accumulate data parsed from multicast data packets and store them in a cache area; each data parsing dynamic library of the device message information collection module uses each delegation method provided by the data parsing dynamic library with the same processing flow.

[0017] According to the seventh data association collection and transmission control method of the first aspect of the present application, an eighth data association collection and transmission control method according to the first aspect of the present application is provided, wherein the self-noise data collection module is connected to the noise monitoring system through the TCP protocol, the self-noise data collection module sends an acquisition control command to the noise monitoring system, after the acquisition starts, the self-noise sampling data packet sent by the noise monitoring system is cyclically received at a specified frequency of once per second, after the acquisition is completed, the acquisition end instruction is sent to the noise monitoring system, and the self-noise data acquisition module closes the self-noise data acquisition thread.

[0018] According to the eighth data association acquisition and transmission control method of the first aspect of the present application, a ninth data association acquisition and transmission control method according to the first aspect of the present application is provided, wherein the array element domain data acquisition and parsing submodule sends an instruction to the array element domain data preprocessing submodule to request the device status. If the device is operating normally and array element domain data acquisition can be performed, an array element domain data acquisition start instruction is sent to the array element domain data preprocessing submodule. After receiving the data acquisition start instruction, the array element domain data preprocessing submodule starts to collect ship array element domain data, and at the same time, the collected array element domain data is subpacketized to generate array element domain data subpackets; the size of the array element domain data subpackets sent by the array element domain data preprocessing submodule is smaller than the amount of data that can be carried by the maximum transmission unit of the link layer.

[0019] According to the ninth data association acquisition and transmission control method of the first aspect of the present application, a tenth data association acquisition and transmission control method according to the first aspect of the present application is provided, wherein the microsecond timer is used for timing with an accuracy of 1 microsecond, and the microsecond timer is triggered when the clock of the CPU core exclusively occupied by the microsecond timer reaches the number of clock cycles of the CPU core within 1 microsecond; time is obtained from the timing device 40 times per second and the local time system information is updated with the obtained time; and sonar target tracking information is obtained 10 times per second; and the data packaging and sending module obtains audio data packets from the buffer at 40ms intervals and sends them over the network.

[0020] According to the tenth data association acquisition and transmission control method of the first aspect of the present application, an eleventh data association acquisition and transmission control method according to the first aspect of the present application is provided, wherein the device message information acquisition module, the audio data acquisition module, the self-noise data acquisition module, the array element domain data acquisition and analysis submodule and the scene data acquisition module are each List <t>Generic as a buffer container to store each data packet into the buffer, and also write to the List being operated when writing to the buffer <t>The buffer is locked and released after the operation is completed; when the data packaging and sending module reads the buffer, the List being operated <t>The buffer is locked and released after the operation is completed.

[0021] According to the second aspect of the present application, a communication device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the first to eleventh configured correlation acquisition and transmission control methods based on underwater acoustic data according to the first aspect of the present application are implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0023] Figure 1 A schematic diagram of a data association acquisition system according to an embodiment of the present application is shown.

[0024] Figure 2 A schematic diagram of a configured correlation acquisition system based on underwater acoustic data according to an embodiment of the present application is shown.

[0025] Figure 3 The processing flow chart of the device message information collection module according to the embodiment of the present application is shown.

[0026] Figure 4 The processing flow chart of the audio data acquisition module according to an embodiment of the present application is shown.

[0027] Figure 5 The processing flow chart of the scene data acquisition module according to an embodiment of the present application is shown.

[0028] Figure 6 The figure shows a processing flow chart of the self-noise data acquisition module according to an embodiment of the present application.

[0029] Figure 7 The figure shows a processing flow chart of the array element domain data acquisition module according to an embodiment of the present application.

[0030] Figure 8 A schematic diagram showing the process of configuration data association according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0031] The following is a clear and complete description of the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0032] Figure 1 A schematic diagram of a data association acquisition system according to an embodiment of the present application is shown.

[0033] The data association acquisition system according to an embodiment of the present application is a device applied to ships, including, for example, a shipborne data acquisition / computing platform, a noise monitoring system, a video acquisition device such as a camera, a sonar, an equipment information acquisition unit, and a network device. The sonar collects underwater acoustic data and provides it to the shipborne data acquisition / computing platform. The shipborne data acquisition / computing platform operates the sonar to collect underwater acoustic data, and sends it to the network device of other nodes (such as a command platform, a shore-based node and / or other ships) through a network link through a network device. The underwater acoustic data obtained by the sonar includes sonar target data and sonar target information. Sonar target data refers to the sonar target audio data and its annotation information obtained from the beam domain, as well as the modulation spectrum data, low-frequency line spectrum data, time-frequency data, and feature data obtained by synchronous processing. The sampling rate of the sonar target audio data is, for example, 44100 Hz.

[0034] The noise monitoring system collects ship noise data. Connected to the shipborne data acquisition / computing platform via a network, the noise monitoring system collects and transmits noise data based on control commands from the platform. The noise monitoring system collects and transmits noise data to the platform at a frequency of, for example, once per second.

[0035] The video capture equipment includes multiple cameras located at designated locations on the ship and connected to the shipboard data acquisition / computing platform via a network. The video capture equipment provides video data to the shipboard data acquisition / computing platform via a transport protocol such as RTSP (Real-Time Streaming Protocol), providing multiple channels of video data.

[0036] The device information collection unit includes multiple devices for acquiring various types of ship information. In various embodiments, multiple information collection units are provided, each using different communication protocols to provide various types of device message information to the shipborne data collection / computing platform. The shipborne data collection / computing platform needs to support communication with each device information collection unit using different communication protocols. The device information collection unit provides, for example, ship service data. Service data includes time information, navigation information, and a timestamp associated with the time information as configuration data. Navigation information includes the current ship's location and ship environment information. Sonar data is also used to obtain sonar target tracking information. Sonar target tracking information is data generated during the acquisition of sonar target tracking.

[0037] It can be understood that each of the various nodes also includes a network device and a computing device.

[0038] According to embodiments of the present application, a shipboard data acquisition / computing platform controls the collection of underwater acoustic data and various other data. For example, the processor of the shipboard data computing / acquisition platform executes software to operate various devices and collect data. Optionally, software running on other nodes collaboratively implements the underwater acoustic data correlation collection method according to embodiments of the present application.

[0039] In view of the diverse nature of ship data and the different interface standards, according to the embodiments of this application, a modular approach is adopted to use different data interfaces and collection methods to collect various types of underwater acoustic configuration data, and the data is reorganized and sorted through multiple pre-processing methods. Different types of data also have different characteristics such as transmission frequency, data bandwidth (the amount of data generated per unit time), and data continuity. When collecting these data concurrently or simultaneously, it is necessary to implement the collection based on the characteristics of each type of data and establish associations between the data to facilitate subsequent data utilization.

[0040] Figure 2 A schematic diagram of a configured correlation acquisition and transmission system based on underwater acoustic data according to an embodiment of the present application is shown.

[0041] See Figure 2 As an example, the software running on the shipborne data acquisition / computing platform forms a configured associated acquisition and transmission system based on underwater acoustic data according to an embodiment of the present application.

[0042] The configurable, correlated acquisition system for underwater acoustic data includes a device message information acquisition module, an audio data acquisition module, a scene data acquisition module, a self-noise data acquisition module, an array element domain data acquisition module, a timing module, a timer module, a buffer, and a data packaging and transmission module. For example, each of these modules, except the buffer, is implemented as one or more threads. Alternatively, these modules can be implemented by software located on different nodes, with inter-module communication achieved via a network.

[0043] The device message information collection module is used to obtain the device information collection unit (also see Figure 1 ) provides device message information. In order to support a variety of device information collection units and device message information message formats, a corresponding dynamic library (called a data parsing dynamic library) is provided for each device information collection unit. Figure 2 In the example, the device message information collection module includes three dynamic libraries (data parsing dynamic library 1, data parsing dynamic library 2, and data parsing dynamic library 3). To collect device messages, different data parsing dynamic libraries are dynamically loaded for different devices (and message formats).

[0044] The audio data acquisition module is used to collect audio data, such as from sonar sources. Three methods exist for collecting audio data: sound card acquisition, audio acquisition card acquisition, and UDP multicast acquisition. For example, ships may have a variety of sonar devices, each with different interfaces, connected to different audio devices on the shipboard data acquisition / computing platform. Therefore, the audio data acquisition module according to embodiments of the present application acquires audio data through various audio data acquisition methods. Different audio data acquisition methods are used, for example, through different threads.

[0045] The scene data acquisition module is used to collect scene data, which is the real-time video data displayed by the ship's station. This module collects scene data by connecting to the ship's network. This module includes, for example, the FFmpeg codec library for encoding and decoding video data collected via the RTSP protocol, and supports parallel acquisition of multiple channels of video data. For each channel of video data, a corresponding thread is created for acquisition.

[0046] The self-noise data acquisition module is used to collect the ship's self-noise data. The self-noise data acquisition module is connected to the noise monitoring system via a wired or wireless network (see also Figure 1 The frequency of sending sub-noise data is 1 second per time. To avoid affecting the collection of other data when there is no data transmission, a dedicated data collection thread is created to collect self-noise data.

[0047] The array element data acquisition module is used to collect ship array element data. It includes an array element data preprocessing submodule and an array element data acquisition and parsing submodule. These two submodules are deployed on different information processing devices and connected via a network. The array element data preprocessing submodule uses LVDS (Low-Voltage Differential Signaling) transmission technology to collect and read ship array element data. After data acquisition, it packages the array element data and sends the array element data packets. The array element data acquisition and parsing submodule creates a data acquisition thread to receive and parse the data packets.

[0048] The data collected by the device message information acquisition module, audio data acquisition module, scene data acquisition module, self-noise data acquisition module, and array element domain data acquisition module are each stored in a buffer. The data units stored in the buffer are called data packets. The data packets generated by these modules are of different sizes, and the data packets of each module do not need to be synchronized with each other (the number of data packets generated by each module in the same time period does not need to be the same, and the time at which each module stores data packets in the buffer does not need to be the same).

[0049] The data packaging and sending module takes one or more data packets from the buffer and sends them to the data user through the network, or stores the data packets on a recording medium. The recipient of the data sent by the data packaging and sending module may be located at, for example, another node.

[0050] The data packaging and transmission module is a consumer that obtains data from the buffer, while the device message information acquisition module, audio data acquisition module, scene data acquisition module, self-noise data acquisition module, and array element domain data acquisition module are each producers that store data in the buffer. The producer and consumer threads share the buffer and operate independently, allowing them to operate in parallel. For example, while the audio data acquisition module is storing audio data in the buffer, the data packaging and transmission module is reading or sending self-noise data from the buffer. Optionally, a locking mechanism is implemented to prevent producer and consumer threads from accessing the same data simultaneously.

[0051] Time information serves as a unified timestamp to synchronize various configuration data. The accuracy of time plays a crucial role in the accuracy of the association of configuration data. The timing module collects data from the timing device 40 times per second and uses the collected time as the local timestamp. When adding a data packet to the buffer, the current local timestamp is appended to the data packet as the time information of the data packet. Sonar target tracking information is also obtained 10 times per second as local service information, and local service data is updated. Sonar target tracking information is obtained from, for example, analyzing sonar target audio data. The local timestamp and local service information are used to establish associations between multiple data. This allows the local timestamp to form a unified timeline for each audio data, and the corresponding audio data can be retrieved on the unified timeline based on its sonar target tracking information through the sonar target data.

[0052] The timer module is used to provide accurate triggering time for the data packaging and sending module. To ensure that the user or recipient can listen to the audio data in real time, the timer module is triggered every 40ms, and in response, the data packaging and sending module sends a sonar target audio data packet. Ordinary timers have a time error, and the error time range is about 10ms, which will greatly affect the transmission frequency and audio listening effect. According to the embodiment of the present application, the timer module is triggered with microsecond accuracy. The timer obtains the system performance frequency of the current computer (for example, the processor clock frequency, in Hz) and calculates the system performance counting frequency = system performance frequency / 1000000. The timer module is used as a timer thread and is set to exclusively occupy a certain CPU core to ensure that it can count the processor clock and prevent other threads from preempting the timer thread. In the timer thread, for example, the processor clock is counted, and the system performance counting frequency is used as the counting trigger threshold to obtain the microsecond clock. This triggers the execution of the callback function by timing 40,000 microseconds (40ms) through the microsecond clock, achieving the purpose of triggering the timer module every 40ms.

[0053] Figure 3 The processing flow chart of the device message information collection module according to the embodiment of the present application is shown.

[0054] According to embodiments of the present application, the device message information collection module supports multiple device information collection units. Furthermore, even after the device message information collection module is deployed, the corresponding device information collection unit may still change or be updated. A data parsing dynamic library is provided for each device information collection unit, or for each type of information to be collected by each device information collection unit. The device message information collection module loads the corresponding data parsing dynamic library based on the device information collection unit from which data is to be collected or the information to be collected.

[0055] The device information collection unit communicates with the device message information collection module via UDP multicast. The device message information collection module and the information to be collected have a designated multicast address. The device message information collection module collects various information from the device information collection unit via UDP multicast and parses it using a loaded data parsing dynamic library.

[0056] For example, the device information collection unit provides three different types of data, each of which has a specified data format and its own UDP multicast address. Each type of data also has a different sending rhythm (for example, one type of data needs to be collected once per minute, while the other type of data needs to be collected once per minute). The three types of data are collected and parsed by data parsing dynamic library 1, data parsing dynamic library 2, and data parsing dynamic library 3 respectively. The device message information collection module loads the above three dynamic libraries and synchronously collects the three types of data provided by the device information collection unit. According to the different types of data collected, the above three dynamic libraries are added to different UDP multicasts respectively.

[0057] The device message information acquisition module starts a thread for each multicast address to receive data. Optionally, each data parsing dynamic library has a specified receiving beat (corresponding to the sending beat of the corresponding data), and converts the network data packets received according to the UDP protocol into the specified structured data through data format conversion methods such as high-low bit conversion, data type conversion, serialization mapping, etc., and converts binary data into value types. Optionally, the device message information acquisition module also sends the received data according to the receiving beat of the data, or stores the data in a buffer. Still optionally, in order to facilitate expansion, for example, adding a data parsing dynamic library, operations such as obtaining the data to be collected from the device information acquisition unit, sending the collected data, and storing the collected data in the buffer are implemented in a delegated manner, and in the device message information acquisition module, the various delegate methods provided by the data parsing dynamic library are used for a variety of different data parsing dynamic libraries with the same or substantially the same processing flow. For example, the collection rhythm and multicast address are obtained from the loaded data parsing dynamic library, and UDP multicast is added. At the obtained collection rhythm, the method for obtaining the data to be collected, sending the collected data, and storing the collected data in the buffer are sequentially called by delegation to complete the collection of device message information. Still optionally, even if each type of collected data has a different rhythm, the device message information collection module sends the collected data at a unified rhythm or stores the collected data in the buffer. Still optionally or further, the data generated by the device message information collection module also has a unified format to facilitate subsequent processing or storage.

[0058] As an example, the collected data is accumulated to generate data packets (device message packets), using List <t>Generic acts as a buffer container, adding generated data packets to the buffer as generic entries. When generating data packets, the current local timestamp and sonar target tracking information synchronized in real time are also attached to each data packet and stored in the buffer as part of the data packet.

[0059] The device message information collection module can collect data from a single device or from multiple devices at the same time. Optionally, by creating multiple threads, each thread loads a data parsing dynamic library corresponding to the device or data type to collect data from multiple devices at the same time.

[0060] The data parsing dynamic library is stored in a designated location, such as a specified directory in a file system. The device message information collection module retrieves and loads the corresponding data parsing dynamic library based on the information of each device information collection unit. Therefore, when adding or changing a device information collection unit on a ship, it is only necessary to place the new data parsing dynamic library in, for example, a designated directory and provide the device message information collection module with the information (e.g., name) of the device information collection unit from which data is to be collected. The newly added dynamic library is then loaded into the device message information collection module to dynamically expand the collection protocol.

[0061] Figure 4 The processing flow chart of the audio data acquisition module according to an embodiment of the present application is shown.

[0062] Audio data has multiple sources, such as sound card, audio capture card and audio capture device.In different embodiments, these audio data sources may exist simultaneously, or there are one or more of them.

[0063] Sound card acquisition uses NAudio (NAudio is from https: / / github.com / naudio / NAudio The NuGget package of the available audio processing library is used for collection, the MMDeviceEnumerator() function is used to obtain the local sound card input device list, and the sound card input device name is monitored according to the specified sound card input device name. The sampling rate, sampling bit number, and sampling channel number are set for the WaveInEvent() function, and an event function is attached to collect sound card data through the event callback function.

[0064] Audio capture card through SharpPcap (SharpPcap is from https: / / github.com / dotpcap / sharppcap The CaptureDevice class, provided by the available network packet capture tool, traverses information about all local network adapters. This class is a singleton instance and holds a cached list of network adapters. The Description property contains a human-readable name and description. The capture device is opened using the Open() function, and parameters can be configured upon opening the device. There are two capture modes for capturing packets from a network adapter: DeviceMode.Normal and DeviceMode.promiscuous. In DeviceMode.Normal, the network adapter only captures local packets, ignoring packets exchanged by other hosts on the network. DeviceMode.promiscuous is promiscuous mode, capturing all packets on the network. Upon completion of packet capture, the OnPacketArrival event is used to receive the packet. The received packet is parsed, containing information such as the captured packet and its length. The data elements in the packet are parsed according to the protocol used to send the packet, obtaining the sonar target audio data. For example, at a sampling rate of 44100 Hz, each packet contains 100 audio data samples, which is received 441 times per second.

[0065] UDP multicast audio data collection relies on the ship's audio data acquisition equipment. The audio data acquisition module queries the device's status via UDP multicast commands and controls the device by sending start and stop audio acquisition commands via UDP. After triggering the audio data acquisition device, the module joins the designated multicast address via UDP to receive audio data. The audio data is then mapped into value-type data and output via a delegate.

[0066] The audio data collected from various sources are analyzed according to their respective sampling rates, sampling bits and other characteristics, and the audio data samples are extracted. Then, the audio data with the specified format is generated and sent to the user or stored in the buffer. As an example, using List <t>Generics act as a buffer container, adding generated audio data packets to the buffer as generic entries. When generating packets, the current local timestamp synchronized in real time is appended to each packet and stored in the buffer as part of the packet.

[0067] Figure 5 The processing flow chart of the scene data acquisition module according to an embodiment of the present application is shown.

[0068] The scene data acquisition module is responsible for collecting scene data, which is the real-time video (including audio) data displayed on the ship's platform. The scene data acquisition module collects scene data by connecting to the ship's network.

[0069] The scene data acquisition module obtains network video data based on the RTSP transmission protocol and uses the FFmpeg codec library (FFmpeg is a http: / / ffmpeg.org / Available audio and video codec library) to encode and decode the video, using multi-threading technology to support the acquisition of multiple channels of video data at the same time.

[0070] During scene data collection, the scene data acquisition module connects to the ship's network and controls the scene data collection process by sending and receiving message instructions via UDP multicast. Before scene data collection, the scene data acquisition module sends a status request instruction. If the device is operating normally and scene data collection can be performed, it sends a scene data collection start instruction to collect the video data displayed on the ship's platform. The scene data is a video data stream sent via the RTSP protocol. The scene data acquisition module receives and parses the data. The collection and parsing process is as follows:

[0071] 1) Create a video data acquisition thread. Create a new thread for each video acquisition channel, based on the number of channels being collected. Each thread performs video data acquisition. After the thread is successfully created, initialize the relevant parameters of the FFmpeg codec library. If there are multiple channels of video data at the same time, create multiple threads to collect each channel of video data separately.

[0072] 2) Determine the rtsp video stream address through the avformat_open_input() function provided by the FFmpeg codec library, open the rtsp video data stream, and obtain the video stream information through the avformat_find_stream_info() function to obtain the AVFormatContext structure.

[0073] 3) By parsing the nb_streams member variable in the AVFormatContext structure, the number of audio and video data streams in the video data is obtained. According to the enumeration value AVMediaType of the audio and video stream, the encoding type of the data stream can be determined. AVMEDIA_TYPE_VIDEO is the video type, and AVMEDIA_TYPE_AUDIO is the audio type. According to the different encoding types of the audio and video data streams, corresponding parsing is performed respectively.

[0074] 4) After obtaining the data stream, use the avcodec_find_decoder() function provided by the FFmpeg codec library to obtain the decoder corresponding to the data stream, open the corresponding decoder to decode the data stream, use the avcodec_find_encoder() function to obtain the encoder, encode the data stream, and obtain the encoded video stream data and / or audio stream data depending on the data stream type. The data obtained by decoding the audio stream is, for example, in PCM format, and the data obtained by decoding the video stream is, for example, in YUV format.

[0075] The collected scene data may have various formats, and the user end may lack one or more encoders and be unable to use the scene data. To facilitate user use of the data, the scene data collection module according to the embodiment of the present application can optionally re-encode the collected audio and video data streams into a specified unified format.

[0076] 5) After obtaining the re-encoded video stream data or audio stream data, use the av_read_frame() function provided by the FFmpeg codec library to loop through the data stream data packetization data, return the packetization data in the format of the AVPacket structure, and assign values ​​to the member variables pts display timestamp and dts decoding timestamp in AVPacket respectively.

[0077] 6) After the audio and video data packets are edited, use the av_interleaved_write_frame() function provided by the FFmpeg codec library to write the data packets to the output stream and merge the audio and video streams.

[0078] The data packets (scene packets) generated by the combined audio and video streams are sent to the user or stored in the buffer. <t>Generic acts as a buffer container, adding generated data packets to the buffer as generic entries. When generating data packets, the current local timestamp and sonar target tracking information synchronized in real time are also attached to each data packet and stored in the buffer as part of the data packet.

[0079] 7) After the scene data collection is completed, close the corresponding data collection thread.

[0080] After the scene data acquisition module finishes collecting data, it sends a scene data collection end instruction, and the scene data collection is completed.

[0081] Figure 6 The figure shows a processing flow chart of the self-noise data acquisition module according to an embodiment of the present application.

[0082] The self-noise data acquisition module is responsible for collecting the ship's self-noise data. The module is connected to the ship's network via a network cable and to the ship's noise monitoring system via the TCP / IP protocol to collect the self-noise data. Since TCP is a reliable transmission method, the integrity of the data is guaranteed.

[0083] During the self-noise data collection process, the self-noise data collection module interacts with the noise monitoring system by sending and receiving message instructions via TCP communication to control the collection and reception of self-noise data. After connecting to the noise monitoring system, the self-noise data collection module sends collection control commands to it. In response, the noise monitoring system begins collecting self-noise data and continuously sends it to the self-noise data collection module, for example, once every second. Optionally, due to the relatively low frequency of self-noise data transmission, the sub-noise data collection thread sleeps after receiving the self-noise collection data packet to reduce system resource usage. The sleep duration coincides with the frequency at which the noise monitoring system sends self-noise data.

[0084] The self-noise data acquisition module receives the noise data sent by the noise monitoring system and analyzes it. The processing flow of the self-noise data acquisition module is as follows:

[0085] Create a self-noise data collection thread, initialize a TCP connection, and connect to the noise monitoring system using the specified IP address and port number. Once connected, send a data collection start command to the noise monitoring system. Once collection begins, receive the self-noise data packets sent by the noise monitoring system.

[0086] The acquisition thread of the self-noise data acquisition module receives the self-noise sampling data packet in a loop, parses the self-noise sampling data packet, and obtains the self-noise data from it. Each sample of the self-noise data is a two-byte short type (short integer) and is stored as a byte stream. The sample of the self-noise data sent by the noise monitoring system is an electrical signal conditioned by the AD (analog-to-digital conversion) board and the conditioning board. The sample of the self-noise data needs to be converted. The conversion method is

[0087]

[0088] Where a is the sensor sensitivity, b is the conditioning gain, c is the A / D gain, and x is the two-byte sample of short self-noise data returned.

[0089] After data analysis and conversion are completed, self-noise data is obtained.

[0090] After the self-noise collection is completed, the self-noise data collection module sends a collection end instruction to the noise monitoring system, the noise monitoring system stops collecting, and the self-noise data collection module closes the corresponding data collection thread.

[0091] Generate a data packet with a specified format for the self-noise data and send it to the user or store it in the buffer. As an example, use List <t>Generic acts as a buffer container, adding generated data packets to the buffer as generic entries. When generating data packets, the current local timestamp and sonar target tracking information synchronized in real time are also attached to each data packet and stored in the buffer as part of the data packet.

[0092] Figure 7 The figure shows a processing flow chart of the array element domain data acquisition module according to an embodiment of the present application.

[0093] The array element data acquisition module is responsible for collecting ship array element data. It consists of an array element data preprocessing submodule and an array element data acquisition and parsing submodule. These two submodules are deployed on different hardware devices and connected via a network. The array element data preprocessing submodule collects ship array element data using LVDS transmission technology. After data acquisition, it encapsulates the array element data and sends the array element data packets via UDP multicast to the array element data acquisition and parsing submodule. The array element data acquisition and parsing submodule receives and parses the array element data packets, verifying the packet header and sequence number to ensure data integrity.

[0094] During the array element domain data collection process, the array element domain data collection and analysis submodule sends and receives message instructions through UDP multicast to communicate and interact with the array element domain data preprocessing submodule. Before the array element domain data is collected, the array element domain data collection and analysis submodule sends an instruction to the array element domain data preprocessing submodule to request the device status. If the device is operating normally and array element domain data collection can be performed, the array element domain data collection start instruction is sent to the array element domain data preprocessing submodule. After receiving the data collection instruction, the array element domain data preprocessing submodule starts to collect the ship array element domain data and at the same time subpackages the array element domain data. The size of the array element domain data subpacket is, for example, 1480 bytes, which is smaller than the amount of data that the link layer MTU (maximum transmission unit) can carry, to ensure the efficiency of data transmission, and the array element domain data subpacket is sent through UDP multicast. The array element domain data collection and analysis submodule receives the data and parses it. The collection and analysis process is as follows:

[0095] (1) Create an array element domain data collection thread, create a UDP connection, bind the multicast address and port number corresponding to the array element domain data, and monitor the data.

[0096] (2) Due to the large amount of array element domain data and the large number of packets, a do-while loop is used in the thread to continuously receive and read the socket multicast array element domain data packets to ensure receiving efficiency.

[0097] (3) Parse the data packet according to its message format, verify the packet header and sequence number, and ensure the integrity of the data.

[0098] (4) According to the data packet format, the array element domain data in the array element domain data packet is obtained, and the array element domain data buffer is set to accumulate data. After the accumulation of the array element domain data is completed, it is forwarded to reduce the number of forwarded packets and reduce the forwarding pressure. Generate a data packet with a specified format for the array element domain data and store it in the buffer. As an example, using List <t>Generic acts as a buffer container, adding generated data packets to the buffer as generic entries. When generating data packets, the current local timestamp and sonar target tracking information synchronized in real time are also attached to each data packet and stored in the buffer as part of the data packet.

[0099] (5) After the array element domain data collection is completed, the corresponding data collection thread is closed.

[0100] After the array element domain data acquisition module completes the acquisition, the array element domain data acquisition and analysis submodule sends an array element domain data acquisition end instruction. After receiving the instruction, the array element domain data preprocessing submodule ends the data acquisition, and the array element domain data acquisition is completed.

[0101] The above describes the process of collecting various types of configuration data on a ship. According to an embodiment of the present application, in addition to collecting data, multiple types of data are synchronously associated so that when the data is subsequently used, the associated multiple types of data are obtained to facilitate data processing, analysis, and utilization.

[0102] Figure 8 A schematic diagram showing the process of configuration data association according to an embodiment of the present application is shown.

[0103] According to an embodiment of the present application, in order to reflect the association between multiple data and facilitate the comprehensive utilization of multiple data, the video data, self-noise data, and array element domain data are synchronously associated with service data as the main line and sonar target data as the core.

[0104] Service data includes time-based information, navigation information, and sonar target tracking information. Time-based information includes timestamps used to unify configuration data. Navigation information includes the current ship or node's location and environmental information. Sonar target tracking information is the data generated during the sonar tracking process. Service data is the primary source to ensure a unified time baseline.

[0105] The collected data comes from a variety of devices, and the sampling times of various data vary. However, when applying the data, it is necessary to analyze the various data based on their timing. For example, to process an audio signal, it is also necessary to obtain self-noise data at the same time. Therefore, the audio signal and the self-noise data need to have associated timestamp information. According to the embodiments of the present application, the time information is used as a unified timestamp to synchronize the various data, so that the collected data has unified and consistent time information.

[0106] Time accuracy is crucial to the accuracy of configuration data association. To this end, the local timestamp is updated by acquiring time information provided by the timing device in real time. This local timestamp is then used to attach timing information to various collected data. For example, the timing device is collected 40 times per second (every 25 milliseconds) and the collected results are synchronized to the local timestamp.

[0107] Furthermore, sonar target tracking information is used to correlate various data. This information is collected 10 times per second (once every 100 milliseconds) and synchronized locally, ensuring consistency between service data and locally updated information.

[0108] The sonar target data obtained through sonar is the core data of the configuration data association. The sonar target data includes sonar target audio data and target direction data. In order to associate the sonar target data, when the sonar target data is collected through the network, local time information is also added and stored in the buffer. As previously disclosed, the time information is updated in real time using the timing device. When storing the sonar target data and time information in the buffer, the List <t>Generic as a buffer container, the buffer size is adaptively set according to the sampling rate of the sonar target audio data. <t>Generic buffers can avoid data boxing and unboxing operations when inserting or reading data, greatly improving the real-time performance of data operations.

[0109] Read data from the buffer and send it to the user or other nodes. When the buffer is read and written at the same time, index error or index overflow abnormal operation will occur. In order to solve this problem, when reading and writing the buffer, the List being operated <t>The buffer is locked and the lock is automatically released after the operation is completed.

[0110] To ensure real-time audio data listening, the sonar target audio data stored in the buffer is sent to users or other nodes at a rate of 25 packets per second, or every 40ms (to accommodate network processing delays). A timer that expires every 40ms is set to trigger the periodic reading of buffer data. According to an embodiment of the present application, a custom microsecond timer is used to provide a trigger function that fires every 40ms.

[0111] The self-noise data association method is based on service data and sonar target data, and multiple channels of self-noise data are collected through the network. The number of channels can be increased or decreased according to the collection requirements. When collecting self-noise data, parameters such as sonar model, target number, and number of channels are received. After collecting self-noise data, a thread is dynamically created based on the number of channels to cache and accumulate each channel of data after unpacking. The collected self-noise data is added to the buffer. The buffer uses List <t>Generic. Based on a pre-set time frequency, a custom microsecond timer is used to periodically retrieve data from each thread to ensure consistent transmission and reception frequencies for all types of data. When adding self-noise data to the buffer, unified timing information and sonar target tracking information are also appended, forming a unified timeline through timestamps. On this unified timeline, the corresponding self-noise data can be retrieved based on the sonar target tracking information of the sonar target data, thereby linking the self-noise data with the sonar target data. Optionally, unified timing information and sonar target tracking information are appended when retrieving and sending self-noise data from the buffer.

[0112] The video data association method is based on service data and sonar target data, and collects multiple video data through the network. After receiving the encoded video data, a thread is dynamically created to accumulate data. The buffer uses List <t>Generic. When adding or removing video data from the buffer, unified timing information and sonar target tracking information are appended, forming a unified timeline through timestamps. The corresponding video data can be retrieved from the sonar target data on the unified timeline.

[0113] The method of associating element domain data is to dynamically create threads according to the acquisition path parameters when collecting element domain data, and associate the element domain data with the sonar tracking target and time information through the current sonar target tracking information and time information. Each thread receives the element domain data and stores it in the buffer. After receiving the collected element domain data, the List <t>Generics serve as a buffer container, storing element-domain data in a custom-sized cache. Based on a pre-set frequency, a custom microsecond timer periodically retrieves cached data from each thread to generate element-domain data packets, ensuring consistent transmission and reception frequencies for all types of data. When adding or removing element-domain data from the buffer, timing information and sonar target tracking information are added to the element-domain data packets. This allows the tracing or retrieval of service data and, in turn, sonar target data using timestamps and sonar target tracking information, enabling data association with service data as the primary focus and sonar target data as the core.

[0114] Return to view Figure 2 When the configuration-based association acquisition system for underwater acoustic data according to the embodiment of the present application works, one or more data in the configuration data are in a unified structure (List <t>Generic) is added to the buffer, and various data are each attached with time information and / or sonar target tracking information. The data packaging and sending module delivers one or more associated data to the user or other nodes in a unified direction. Optionally, according to demand or user instructions, the data packaging and module delivers one or more specified data. The data packaging and sending module delivers specified types of data at a specified period or frequency in response to the expiration of a custom microsecond timer. For example, when sonar target data needs to be delivered, sonar target data is obtained from the buffer every 40ms and sent to the user or other node. In response to the expiration of the custom millisecond timer, one or more other data (attached with time information and / or sonar target tracking information) are optionally obtained from the buffer and sent to the user or other node. When sending various types of data, each type of data has its own sending period, and the sending periods of various types of data are the same or different.

[0115] Thus, the data sent to users or other nodes by the configured correlation acquisition system based on underwater acoustic data according to an embodiment of the present application is, for example, multiple concurrent data streams of multiple types of data. Each data stream has its own transmission period, and unified timing information is attached to each data packet of the multiple data streams, allowing users to use timestamps to obtain data associated with a specified time or time period in each type of data (for example, obtaining sonar target audio data, self-noise data, and array element domain data at the same time point t from each received data stream). One or more data streams are also attached with unified sonar target tracking information, so that data associated with a specified sonar target in each type of data can be obtained based on the sonar target tracking information.

[0116] Furthermore, the user or node receiving the data obtains multiple types of data in a unified way. <t>Generic) is added to and removed from the buffer, eliminating the need for users or nodes to worry about the different ways and formats of collecting different types of data. Each type of data is provided to users or receiving nodes through the network as an independent data stream, allowing them to receive each data stream separately and choose to receive or not receive one or more data streams.

[0117] In some cases, one or more types of data are continuously generated and collected, while the user or receiving node chooses not to receive one or more types of data. In an optional embodiment, the buffer is set as a circular queue, and the collected data is continuously added to the circular queue corresponding to its type. If the existing data is not taken out of the circular queue and sent to the user or receiving node, after the queue is filled, the newly collected data overwrites the old data. Therefore, even if the user or receiving node does not receive one or more types of data, it will not affect the collection of one or more types of data, and it will not affect the establishment of associations between one or more types of data, and when the user needs to receive data, the latest data or data associated with other types can be immediately obtained from the buffer and sent to the user. Optionally, when the data packaging and sending module obtains one or more types of data from the buffer, it obtains data of each type with an association relationship from the buffer based on the time information to avoid the user receiving incomplete data of the type. It is understandable that the collection periods of different types of data may be different. For example, there may be 40 pieces of one type of data in the buffer per second, while there may be 1 piece of another type of data in the buffer per second. Thus, the 40 pieces of the first type of data may be associated with the 1 piece of the second type of data. Thus, the number of pieces of data of different types that are associated may be different.

[0118] Through one or more embodiments of the present application, various types of discrete data are collected and integrated into structured data, thereby achieving the association of various types of data and promoting the processing and further utilization of various types of data.

[0119] The configuration-based, correlated acquisition system for underwater acoustic data provided in the embodiments of the present application includes, for example, a computer, server, or other information processing device. These devices include, for example, memory, one or more processors, one or more presentation components, I / O components, and a power supply, all directly or indirectly coupled to a bus. The bus can represent one or more types of buses (e.g., an address bus, a data bus, or a combination thereof).

[0120] Although preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they are aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of this application. Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if such changes and modifications fall within the scope of the claims of this application and their equivalents, then this application is intended to include such changes and modifications.< / t> < / t> < / t> < / t> < / t> < / t> < / t> < / t> < / t> < / t> < / t> < / t> < / t> < / t> < / t> < / t>

Claims

1. A data association collection and transmission control method, comprising: At specified time intervals, the system obtains time from the timing device and uses the obtained time to update the local time information. Get sonar target tracking information at specified time intervals; The device message information collection module loads one or more data parsing dynamic libraries corresponding to the device information collection unit, creates threads corresponding to the data types according to the one or more data types provided by the device information collection unit, adds a multicast group corresponding to the data type to each created thread, and uses the first data parsing dynamic library corresponding to the data type of the one or more loaded data parsing dynamic libraries to receive and parse the multicast data packet according to its own data parsing method to obtain the first data provided by the device information collection unit, accumulates one or more copies of the first data to generate a device message data packet, appends the current local time system information to the generated device message data packet, and stores the generated device message data packet in a buffer; The audio data acquisition module collects one or more types of audio data from a sound card, an audio capture card, and / or an audio capture device, accumulates one or more copies of the same type of audio data to generate an audio data packet, appends the current local time information to the generated audio data packet, and stores the resultant audio data packet in a buffer; The self-noise data acquisition module creates a self-noise data acquisition thread, receives self-noise sampling data packets sent from the noise monitoring system at a specified time interval based on a specified network address in the created self-noise data acquisition thread, obtains self-noise data samples from the self-noise sampling data packets, converts the self-noise data samples to obtain self-noise data; accumulates the self-noise data to generate a self-noise data packet; appends the current local time information and the current sonar target tracking information to the generated self-noise data packet and stores it in a buffer; The array element domain data acquisition and parsing submodule creates an array element domain data acquisition thread. In the array element domain data acquisition thread, it cyclically receives array element domain data packets sent by the array element domain data preprocessing submodule from a specified multicast address, accumulates array element domain data from the array element domain data packets to generate array element domain data packets; appends current local time information and current sonar target tracking information to the generated array element domain data packets and stores them in a buffer; The data packaging and sending module obtains device message data packets, audio data packets, self-noise data packets and / or array element domain data packets from the buffer at one or more time intervals, and sends them through the network; The timer thread that exclusively occupies the CPU core counts the clock of the CPU core to obtain a microsecond clock, and the one or more time intervals are timed by the microsecond clock. In response to the expiration of the one or more time intervals, the data packaging and sending module is triggered to obtain one or more of the device message data packet, audio data packet, self-noise data packet and array element domain data packet, and send them through the network.

2. The method according to claim 1, further comprising: The scene data acquisition module creates a corresponding thread for each video to be collected, obtains audio data stream and video data stream based on the RTSP protocol in the created thread, decodes the obtained audio data stream and video data stream to obtain audio stream data and video stream data respectively, encodes the audio stream data and video stream data into a specified format, merges the encoded audio stream data and video stream data to generate a scene data packet; appends the current local time system information and the current sonar target tracking information to the generated scene data packet and stores it in a buffer; The data packaging and sending module obtains the scene data packet from the buffer at a specified time interval and sends it through the network; wherein the data packaging and sending module is triggered to obtain the scene data packet and send it through the network in response to the expiration of the specified time interval of the microsecond clock timing.

3. The method according to claim 2, wherein Before collecting scene data, the scene collection module sends a collection start instruction to collect scene data; for multiple videos existing at the same time, multiple corresponding threads are created to collect each video separately; the video stream address is obtained through the avformat_open_input() function provided by the FFmpeg codec library, and the video stream information is obtained through the avformat_find_stream_info() function to obtain the AVFormatContext structure; by parsing the nb_streams member variable in the AVFormatContext structure, the number of audio data streams and video data streams is obtained, and the encoding type of the data stream is determined according to the enumeration value AVMediaType; for the encoded audio stream data and video stream data, the av_read_frame() function is used to loop to obtain the data stream data packet data, and return the packet data in the format of the AVPacket structure; the av_interleaved_write_frame() function is used to merge the audio stream data and the video stream data.

4. The method according to claim 1, wherein The self-noise data acquisition module uses the formula to sample the self-noise data The self-noise data is obtained by conversion, where a is the sensor sensitivity, b is the conditioning gain of the conditioning board of the noise monitoring system, c is the A / D gain of the AD board of the noise monitoring system, and x is the sampling of the self-noise data.

5. The method according to claim 4, wherein For the sound card, the audio data acquisition module monitors the sound card input device to collect audio data from the WaveInEvent event; for the audio capture card, the audio data acquisition module captures the data packets provided by the audio capture card from the network adapter and parses the audio data according to the audio sampling rate and data length; for the audio capture device, the audio data acquisition module receives audio data from the multicast address corresponding to the audio capture device, and also queries the status of the audio capture device through UDP multicast commands, and sends audio start and stop capture instructions to control the audio capture device to send or stop sending audio data.

6. The method according to claim 5, wherein The microsecond timer of the timer thread times 40ms time intervals to trigger the collection and transmission of audio data packets from the buffer area.

7. The method according to claim 6, wherein In response to the use of a new device information collection unit, the device message information collection module loads a new data parsing dynamic library corresponding to the data type provided by the new device information collection unit; the loaded data parsing dynamic library uses a delegation method to receive and parse multicast data packets, and accumulates data parsed from multicast data packets and stores them in a cache area; each data parsing dynamic library of the device message information collection module uses the delegation methods provided by the data parsing dynamic library with the same processing flow.

8. The method according to claim 7, wherein The array element domain data acquisition and analysis submodule sends an instruction to the array element domain data preprocessing submodule to request the device status. If the device is operating normally and array element domain data acquisition can be performed, the array element domain data acquisition start instruction is sent to the array element domain data preprocessing submodule. After receiving the data acquisition start instruction, the array element domain data preprocessing submodule starts to collect the ship array element domain data and simultaneously subpackages the collected array element domain data to generate array element domain data subpackages. The size of the array element domain data subpacket sent by the array element domain data preprocessing submodule is smaller than the amount of data that can be carried by the link layer maximum transmission unit.

9. The method according to claim 8, wherein The device message information acquisition module, audio data acquisition module, self-noise data acquisition module, array element domain data acquisition and analysis submodule and scene data acquisition module are each based on List <t>Generic as a buffer container to store each data packet into the buffer, and also write to the List being operated when writing to the buffer <t> The buffer is locked and released after the operation is completed;< / t> < / t> When the data packaging and sending module reads the buffer, it <t> The buffer is locked and released after the operation is completed.< / t> 10. A communication device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method according to any one of claims 1 to 9 is implemented.

Images