Downhole intelligent audio and video communication method and device based on time sensitive network

Abstract

The invention provides an underground intelligent audio and video communication method and device based on a time-sensitive network, and relates to the technical field of communication and the field of information processing.The method comprises the steps that time synchronization is carried out on first-type underground equipment and second-type underground equipment, the first-type underground equipment is used for collecting data, and the second-type underground equipment is used for collecting data; the second type of underground equipment is used for receiving and processing data; a timestamp is given to the to-be-processed data collected by the first type of underground equipment, and the second type of underground equipment carries out data synchronization according to the timestamp; and enabling the second type of underground equipment to process the audio data or the video data in the to-be-processed data. Time synchronization is carried out on the first type of underground equipment and the second type of underground equipment, synchronous processing of the to-be-processed data is achieved, data processing errors caused by data transmission delay are avoided, meanwhile, data synchronization can be achieved, a brand new synchronous processing method can be adopted, and the data processing efficiency is improved. And the communication reliability and the availability of a data processing result are improved.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to an underground intelligent audio and video communication method, device, electronic device and storage medium based on a time-sensitive network. Background Art

[0002] With the development and maturity of real-time Ethernet and 5G technology, the combination of time-sensitive networks (TSN, Time Sensitive Networking) and wireless sensor networks (WSN, Wireless Sensor Network) has gradually become a necessary means of information fusion. In the narrow and long underground tunnels and special communication environments, wired data transmission cannot be flexibly installed in dangerous special areas, and wireless transmission is difficult to ensure information reliability. The traditional coal mine underground safety monitoring and control data transmission method has large delay and low reliability, especially in the working area with high real-time requirements. The low time synchronization accuracy will cause major engineering accidents. Summary of the invention

[0003] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0004] To this end, the first purpose of this application is to propose an underground intelligent audio and video communication method based on a time-sensitive network to achieve.

[0005] The second object of the present application is to provide a device.

[0006] The third objective of the present application is to provide an electronic device.

[0007] A fourth objective of the present application is to provide a computer-readable storage medium.

[0008] A fifth object of the present application is to provide a computer program product.

[0009] To achieve the above-mentioned purpose, the first embodiment of the present application proposes an underground intelligent audio and video communication method based on a time-sensitive network, including:

[0010] Performing time synchronization on at least one downhole device of the first type and at least one downhole device of the second type, wherein the downhole device of the first type is used for collecting data and the downhole device of the second type is used for receiving and processing data;

[0011] Assigning a timestamp to the data to be processed collected by the first type of downhole equipment, and synchronizing the data to be processed according to the timestamp in the second type of downhole equipment;

[0012] The second type of downhole equipment is controlled to process the audio data or video data in the data to be processed.

[0013] Optionally, the first type of downhole equipment is a data acquisition terminal.

[0014] Optionally, the second type of downhole equipment is any one of a data receiving end, a data control end, and a server end.

[0015] Optionally, the method further includes:

[0016] The first type of downhole equipment and the second type of downhole equipment are time calibrated respectively.

[0017] Optionally, the processing of the audio data or video data in the data to be processed includes:

[0018] Using a microphone array enhancement algorithm to process the audio data acquired by the multi-channel acquisition end to improve the audio quality;

[0019] The feature data of the audio data is extracted using a direction of arrival estimation algorithm and an acoustic array algorithm to determine feature information corresponding to the audio data, wherein the feature information includes a sound source location and a sound event category.

[0020] Optionally, the processing of the audio data or video data in the data to be processed includes:

[0021] Sorting and splicing the video data according to the sequence of the timestamps;

[0022] Modeling is performed on the spliced ​​video data to generate a three-dimensional video.

[0023] To achieve the above-mentioned purpose, the second embodiment of the present application proposes an underground intelligent audio and video communication device based on a time-sensitive network, including:

[0024] A time synchronization module, used for performing time synchronization between a first type of downhole equipment and a second type of downhole equipment, wherein the first type of downhole equipment is used for collecting data and the second type of downhole equipment processes data;

[0025] A data synchronization module, used for assigning a timestamp to the data to be processed collected by the first type of downhole equipment, and performing data synchronization on the data to be processed according to the timestamp in the second type of downhole equipment;

[0026] A processing module is used to control the second type of downhole equipment to process the audio data or video data in the data to be processed.

[0027] To achieve the above-mentioned purpose, the third aspect of the present application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0028] The memory stores computer-executable instructions;

[0029] The processor executes the computer-executable instructions stored in the memory to implement the method as described in any one of the first aspects.

[0030] To achieve the above-mentioned purpose, the fourth aspect embodiment of the present application proposes a computer-readable storage medium, in which computer-readable storage medium is stored computer execution instructions, and when the computer execution instructions are executed by a processor, they are used to implement the method as described in any one of the first aspects.

[0031] To achieve the above-mentioned purpose, the fifth aspect of the present application proposes a computer program product, which implements any method in the first aspect when executed by a processor.

[0032] The present application provides an intelligent underground audio and video communication method, device, electronic device and storage medium based on a time-sensitive network. By performing time synchronization between a first type of underground equipment and a second type of underground equipment, synchronous processing of the data to be processed is achieved, thereby avoiding data processing errors caused by data transmission delays. At the same time, since data synchronization is achieved, a new synchronization processing method can be adopted, thereby improving the reliability of communication and the availability of data processing results.

[0033] Further expand the typical functions of underground audio and video networks, further expand the application scenarios, and solve more functional problems of underground audio and video networks.

[0034] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0036] Figure 1 A schematic diagram of a flow chart of an underground intelligent audio and video communication method based on a time-sensitive network provided in an embodiment of the present application;

[0037] Figure 2 A schematic diagram of the structure of an underground intelligent audio and video communication device based on a time-sensitive network provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0039] With the popularity of Ethernet, the demand for Ethernet-based multimedia applications is increasing day by day. In 2006, IEEE established the AVB Working Group and formulated a series of new 802.1 technical standards to expand the functions of existing Ethernet, including bandwidth retention, delay limitation and precise clock synchronization, providing high-quality, low-latency, time-synchronized audio and video LAN solutions. In 2012, the AVB Working Group was renamed the TSN Working Group. On the basis of inheriting the existing AVB technology, it further formulated and proposed more feasible technical standards for real-time communication application scenarios, thereby continuing to lead the development of Ethernet technology in the future in the fields of industry and automobiles.

[0040] The existing remote control communication methods in the mine are mainly based on traditional Ethernet, CAN bus and other mature technologies. Problems such as network communication bandwidth, delay, synchronization, etc. often limit the further promotion of intelligent audio and video applications in the mine. This patent uses the underlying principles of time-sensitive network TSN technology and deeply combines it with underground audio and video applications. It significantly improves the quality of underground network communication in terms of bandwidth stability, delay limitation and precise clock synchronization, thereby driving the deep expansion of underground audio and video intelligent processing applications in the intelligent direction.

[0041] To address this problem, the present invention provides an underground intelligent audio and video communication method based on a time-sensitive network. Figure 1 The following is a flow chart of an underground intelligent audio and video communication method based on a time-sensitive network provided in an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0042] Step 101, time synchronization is performed on at least one downhole device of the first type and at least one downhole device of the second type, wherein the downhole device of the first type is used for collecting data, and the downhole device of the second type is used for receiving and processing data.

[0043] In this embodiment, the existing Ethernet-based communication method has problems such as large network jitter, unstable delay, inability to accurately synchronize at the ms level, and unstable network bandwidth. TSN technology has improved this through four aspects: time synchronization, scheduling delay, reliability, and resource management.

[0044] This proposal applies TSN technology to the communication and intelligent processing of underground audio and video sensor systems, significantly improving the stability and processing capabilities of underground audio and video systems, making the entire underground communication network system more stable and reliable, and providing a solid and reliable infrastructure for intelligent coal mining.

[0045] Due to the complex network conditions in the well, the audio and video monitoring and communication networks built on traditional Ethernet often encounter audio and video freezes, large and unstable delays, asynchronous images, and lack of coordination among multiple machines. In order to solve the above problems, the present invention uses the TSN network to synchronize the time of the well equipment, align the time of each well equipment, and facilitate the subsequent processing of data transmitted between the equipment.

[0046] Traditional underground audio and video networks cannot guarantee the synchronous collection arrival time of sensors, resulting in sensors being nearly independent of each other. The underground audio and video network built using TSN technology can ensure a stable time delay between sensors. Therefore, the entire audio and video network can be regarded as a unified sensor network with a stable time delay, thereby extracting more information and achieving more intelligent processing capabilities.

[0047] Optionally, the first type of downhole equipment is a data acquisition terminal, and a large number of them are usually installed underground.

[0048] Optionally, the second type of underground equipment is any one of a data receiving end, a data control end, and a server end, and a plurality of such devices are usually installed underground.

[0049] The first type of downhole equipment is used to collect data, including audio data, video data, etc.

[0050] Step 102, assigning a timestamp to the data to be processed collected by the first type of downhole equipment, and performing data synchronization on the data to be processed according to the timestamp in the second type of downhole equipment;

[0051] In this embodiment, after collecting data, the first type of downhole equipment assigns a timestamp to the data to be processed collected at each moment, and then sends the data to be processed to the second type of downhole equipment.

[0052] In a possible embodiment, the data receiving end receives the data to be processed sent by the first type of downhole equipment.

[0053] In a possible embodiment, the data to be processed is transmitted between multiple second-type downhole devices. Optionally, the data receiving end sends the data to be processed to the data control end and the server end.

[0054] Step 103: Control the second type of downhole equipment to process the audio data or video data in the data to be processed.

[0055] In this embodiment, the second type of downhole equipment is used to process data. Based on the time synchronization of the equipment in the previous step, the time in the first type of downhole equipment and the second type of downhole equipment is synchronized. Based on the timestamp added by the first type of downhole equipment to the data to be processed, the second type of downhole equipment can know the time point when the first type of downhole equipment collects the data to be processed. The timestamp is matched with the time in the second type of downhole equipment, so that the second type of downhole equipment can accurately know the time point corresponding to the audio data or video data in the data to be processed, so as to facilitate the processing of the data to be processed.

[0056] Optionally, the method further includes:

[0057] The first type of downhole equipment and the second type of downhole equipment are time calibrated respectively.

[0058] Optionally, the step 103 processes the audio data or video data in the data to be processed, including:

[0059] Using a microphone array enhancement algorithm to process the audio data acquired by the multi-channel acquisition end to improve the audio quality;

[0060] The feature data of the audio data is extracted using a direction of arrival estimation algorithm and an acoustic array algorithm to determine feature information corresponding to the audio data, wherein the feature information includes a sound source location and a sound event category.

[0061] In this embodiment, a traditional array signal processing method is used for audio signals, including but not limited to applying a direction of arrival estimation algorithm to locate the sound source, applying an array signal enhancement algorithm to reduce noise and improve the signal-to-noise ratio, and applying a feature extraction and information recognition algorithm to monitor the sound source and identify acoustic events.

[0062] Optionally, the step 103 processes the audio data or video data in the data to be processed, including:

[0063] Sorting and splicing the video data according to the sequence of the timestamps;

[0064] Modeling is performed on the spliced ​​video data to generate a three-dimensional video.

[0065] In this embodiment, with respect to the video signal, the synchronized images are spliced ​​and modeled in real time, and the underground field environment can be dynamically reconstructed and presented in real time through VR glasses, VR screens, etc.

[0067] In the disclosed embodiments, acoustic array technology is used to handle underground acoustic event problems: the previous communication technology based on traditional Ethernet cannot regard the entire audio network as a unified whole due to unstable delays. However, by adopting TSN technology, the entire audio network can be regarded as a unified acoustic sensor array, so that array technology can be used to track, locate, predict and identify acoustic events.

[0068] Using multiple synchronous video streams, the motion of moving objects underground is accurately estimated, including movement paths, operating status, etc., and the underground video images are accurately spliced ​​and solid modeled. Then, VR technology is used to dynamically reconstruct and reproduce the underground scene environment in real time.

[0069] At present, voice communication in the fully mechanized mining face requires very high real-time performance. CAN bus transmission is usually used, and Ethernet forwarding will cause voice distortion and delay in speaking. Building an audio communication network based on TSN technology can effectively solve the problems of timeliness insertion and unstable delay, and greatly optimize the existing voice communication network.

[0070] Figure 2 The following is a schematic diagram of the structure of an underground intelligent audio and video communication device based on a time-sensitive network provided in an embodiment of the present application. Figure 2 As shown, the device comprises:

[0071] A time synchronization module 210, used for performing time synchronization between a first type of downhole equipment and a second type of downhole equipment, wherein the first type of downhole equipment is used for collecting data and the second type of downhole equipment processes data;

[0072] A data synchronization module 220, configured to assign a timestamp to the data to be processed collected by the first type of downhole equipment, and synchronize the data to be processed according to the timestamp in the second type of downhole equipment;

[0073] The processing module 230 is used to control the second type of downhole equipment to process the audio data or video data in the data to be processed.

[0074] It should be noted that the aforementioned explanation of the embodiment of the underground intelligent audio and video communication method based on a time-sensitive network is also applicable to the underground intelligent audio and video communication device based on a time-sensitive network of this embodiment, and will not be repeated here.

[0075] In order to implement the above embodiments, the present application also proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method provided by the above embodiments.

[0076] In order to implement the above embodiments, the present application also proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the methods provided by the above embodiments.

[0077] In order to implement the above embodiments, the present application also proposes a computer program product, including a computer program, which implements the methods provided by the above embodiments when executed by a processor.

[0078] The collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this application are in compliance with relevant laws and regulations and do not violate public order and good morals.

[0079] It should be noted that personal information from users should be collected for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. In addition, such collection / sharing should be carried out after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign the agreement / authorization including authorization of relevant user information before the user uses the function. In addition, any necessary steps should be taken to protect and safeguard access to such personal information data and ensure that others who have access to personal information data comply with its privacy policy and procedures.

[0080] The present application is expected to provide an implementation scheme for users to selectively block the use or access of personal information data. That is, the present disclosure is expected to provide hardware and / or software to prevent or block access to such personal information data. Once the personal information data is no longer needed, the risk can be minimized by limiting data collection and deleting the data. In addition, when applicable, such personal information is de-identified to protect the privacy of the user.

[0081] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0082] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0083] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0084] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0085] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0086] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0087] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0088] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. An underground intelligent audio and video communication method based on a time-sensitive network, characterized in that: The following steps are involved: Performing time synchronization on at least one downhole device of the first type and at least one downhole device of the second type, wherein the downhole device of the first type is used for collecting data and the downhole device of the second type is used for receiving and processing data; Assigning a timestamp to the data to be processed collected by the first type of downhole equipment, and performing data synchronization on the data to be processed according to the timestamp in the second type of downhole equipment; The second type of downhole equipment is controlled to process the audio data or video data in the data to be processed.

2. The method according to claim 1, characterized in that The first type of downhole equipment is a data acquisition terminal.

3. The method according to claim 1, characterized in that The second type of downhole equipment is any one of a data receiving end, a data control end, and a server end.

4. The method according to claim 1, characterized in that: The method further comprises: The first type of downhole equipment and the second type of downhole equipment are time calibrated respectively.

5. The method according to claim 1, characterized in that The processing of the audio data or video data in the data to be processed includes: Using a microphone array enhancement algorithm to process the audio data acquired by the multi-channel acquisition end to improve the audio quality; The feature data of the audio data is extracted using a direction of arrival estimation algorithm and an acoustic array algorithm to determine feature information corresponding to the audio data, wherein the feature information includes a sound source location and a sound event category.

6. The method according to claim 1, characterized in that The processing of the audio data or video data in the data to be processed includes: Sorting and splicing the video data according to the sequence of the timestamps; Modeling is performed on the spliced ​​video data to generate a three-dimensional video.

7. An underground intelligent audio and video communication device based on a time-sensitive network, characterized in that: include: A time synchronization module, used for performing time synchronization between a first type of downhole equipment and a second type of downhole equipment, wherein the first type of downhole equipment is used for collecting data and the second type of downhole equipment processes data; A data synchronization module, used for assigning a timestamp to the data to be processed collected by the first type of downhole equipment, and performing data synchronization on the data to be processed according to the timestamp in the second type of downhole equipment; A processing module is used to control the second type of downhole equipment to process the audio data or video data in the data to be processed.

8. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 6 when being executed by a processor.

Images