A multi-source fusion communication system in an emergency situation and a method of using the same

Abstract

The application belongs to the technical field of network communication, and specifically discloses a multi-source fusion communication system under emergency conditions. The system is mainly applied to the multi-person communication scene under the emergency command in sudden disasters, supports various forms of communication and exchange in the system, fuses and switches the information provided by all parties among audio / video / text through monitoring the available broadband resources, acquires the behavior description information of the main characters in the video during the fusion and switching process, cooperates with the GIS position information of the main characters, and realizes timely and accurate forwarding of the information to each terminal under the emergency conditions according to the actual situation. By using the system, the utilization rate of the network can be improved under the condition that the communication network under the emergency condition is poor, and the communication resources are maximized under the premise of guaranteeing the communication effect as much as possible.

Description

Technical Field

[0001] This invention belongs to the field of network communication technology, and mainly relates to a multi-source fusion communication system and its usage method in emergency situations. Background Technology

[0002] my country is a country prone to natural disasters, with earthquakes, floods, typhoons, and other natural calamities occurring frequently. Maintaining communication with the outside world in disaster areas is extremely helpful for obtaining information about the disaster and designing relief plans. However, when regions are hit by disasters, communication equipment and links are often damaged, necessitating the establishment of temporary communication systems to facilitate disaster relief command and control. In severe situations, even reliance on BeiDou satellite communication may be necessary. For example, the invention patent with patent number 202111247238.0, "Image Transmission Method Based on BeiDou-3 Short Message Channel in Cloud-Edge-End," discloses an image transmission method based on BeiDou-3 short message channel in cloud-edge-end. This method involves acquiring image data through a camera array, performing layered processing through an AI processor, independently compressing and encoding the Region of Interest (ROI) or Target of Interest (TOI) in the acquired image, and injecting the block encoding into the message fields of multiple RDSS cards using a serial port module. The data is then transmitted uplink through the BeiDou-3 communication module and forwarded via the ground control center station. Downlink data is sent to the BeiDou-3 module, reassembled in image data processing via a serial module, and finally displayed on the terminal. The beneficial effects are: it will significantly promote the empowerment of various industries by BeiDou-3 narrowband communication; and as an important means of emergency response, BeiDou-3 short messages will play a greater and deeper role in disaster relief and rescue.

[0003] Although communication resources are extremely valuable in disaster-stricken areas, maintaining uninterrupted communication is crucial for ensuring command and control of disaster relief efforts. When resources are sufficient, video conferencing is far more effective than SMS messaging using the BeiDou Navigation Satellite System. However, allocating all resources to video conferencing would hinder other related tasks. Bandwidth allocation is typically handled by the operator of a temporary communication system after its establishment. Operators, recognizing the strategic importance of the disaster relief command center, often allocate significant bandwidth. In reality, however, if the disaster relief command center could further optimize and manage bandwidth resources based on actual needs, overall network performance and utilization could be significantly improved. Summary of the Invention

[0004] In view of this, the present invention provides a method for optimally utilizing communication resources in disaster-like environments by selecting the optimal information transmission format based on the actual equipment and network resources obtained from the communicating parties. This method also ensures that ineffective communication due to information expression switching is avoided during the conversion process.

[0005] To achieve the above objectives, the present invention provides a multi-source fusion communication system for emergency situations, characterized in that it includes multiple communication terminals, at least one of which includes a command client. The command client includes at least an audio / video acquisition device and an information transmission device. The command client is also equipped with a communication service component, which is used to monitor allocated broadband resources and predict broadband usage, wherein broadband usage includes broadband resources occupied by local communication services and broadband resources occupied by other service resources.

[0006] The audio acquisition device includes a speech recognition and conversion component that converts speech into text information;

[0007] The video capture device is equipped with a streaming media processing module for processing streaming media files. This module includes an identity preset value module, which defines the identities of people appearing in the video. The streaming media processing module splits the video into multiple images and audio information, identifies people in the images and associates them with their corresponding expressions, actions, and voices, and then sends the extracted voice portion to a speech recognition and conversion component for conversion.

[0008] It also includes a text information service component, which converts the expressions, actions, and background information recognized in the image into text information, and obtains the text information output by the speech recognition and conversion component, and organizes the text information according to the relationships between the people to form the output text.

[0009] The client is instructed to select the actual output content based on the predicted bandwidth usage.

[0010] Each communication terminal transmits signals containing GIS location information, facilitating tracking and command.

[0011] Preferably, the communication terminal also includes a short message channel support device for the BeiDou satellite system. This support device is connected to the text messaging service component. The short message channel support device for the BeiDou satellite system set in the command client includes at least a BeiDou satellite receiving antenna, a signal amplifier, and a signal modem. Other communication terminals only include a BeiDou satellite receiver. The function of these devices is to transmit and receive signals from the BeiDou satellite system, amplify and demodulate the received signals to ultimately restore the original short message information; and compress and transmit the transmitted signals. The main function of the BeiDou satellite receiving antenna is to receive and transmit signals from the BeiDou satellite system and transmit them to the signal amplifier, making them easier to process. The function of the signal modem is to demodulate the amplified signal and restore it to the original short message information.

[0012] Preferably, the other communication terminals include individual soldier devices and vehicle-mounted trunking terminals. Individual soldier devices are personal walkie-talkies, smartphones, or tablets with communication functions, which are directly connected to the communication system. Vehicle-mounted trunking terminals, under favorable conditions, are communication platforms built on vehicle-mounted devices for use by multiple units. Individual users on the platform access the communication system through vehicle-mounted devices.

[0013] This invention also includes a method for using a multi-source fusion communication system in emergency situations. The system involves directing clients to conduct normal telephone conferences, collecting video and audio information from the conference via audio / video acquisition devices, and having the communication service component monitor allocated broadband resources in real-time during the video conference. Simultaneously, it predicts broadband usage for the next time period based on the needs of other services, thereby calculating the remaining broadband resources for the next time period. The information transmission device initiates communication service renegotiation, and outputs content based on the assessed communication resources. This assessment is performed based on communication resource thresholds, and the collected video and audio information is processed sequentially according to different thresholds as follows:

[0014] Reduce video resolution;

[0015] Reduce overall streaming bitrate;

[0016] Switch to voice access;

[0017] Switch to sending text messages.

[0018] Preferably, the prediction of bandwidth usage for the next time period based on the demand of other services employs a bandwidth monitoring solution from a cloud computing platform when network conditions are good; otherwise, a QoS policy is used. The optimal solution is selected based on the actual network conditions.

[0019] Preferably, the command client requires users to manually input the designated commander and upload the corresponding commander's image before using the image recognition feature. If there are multiple commanders, their command order should be specified. When multiple people appear in the video, the image only recognizes the commander with the highest priority, thus avoiding the generation of excessive invalid data resources.

[0020] Preferably, the text messaging service component captures the speaker's expression, actions, and the objects of those actions, and converts them into text descriptions.

[0021] When sending text messages, the text messaging service component adds corresponding text descriptions of expressions and actions. This overcomes the problem of insufficient information content caused by simply converting speech to text.

[0022] Preferably, during the meeting, if the service quality detection between the terminal and the communication service component detects that the audio and video communication quality is trending towards deterioration, then when a threshold is reached, the communication service component predicts the duration of adjusted broadband resources and selects a processing scheme that can stably maintain the connection for a certain period. Before switching processing schemes, the component provides voice or text prompts for the replacement scheme. This ensures communication continuity while avoiding the negative impacts of switching.

[0023] Preferably, the duration of the single phase is not less than 15 minutes.

[0024] Preferably, when broadband resources are completely lost, information is switched to the short message channel of the BeiDou satellite.

[0025] Because of this technical solution, this invention can utilize existing teleconferencing systems for communication during multi-person conference calls. During instant messaging, the command client simultaneously stores real-time video, audio, or text information resources. Video, voice, and text messages are sent as needed based on actual network conditions, utilizing minimal communication resources to provide the most effective information communication and obtain corresponding instructions. This invention enables front-end and back-end command and control communication through switchable communication methods, while simultaneously allocating communication resources to other necessary services, thereby achieving high-efficiency utilization of network communication resources.

[0026] It should be noted that software for converting video, audio, and text already exists in the prior art. However, for the scenario in which this invention is based, the requirement for immediacy is much higher. Furthermore, existing technologies can only convert the audio portion of video; the ability to analyze video content and incorporate it into text information to improve communication is something that existing technologies lack. Attached Figure Description

[0027] Figure 1 This is a structural diagram of the system in this invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in sequences other than those illustrated or described herein.

[0030] It should be understood that in the various embodiments of the present invention, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0031] It should be understood that in this invention, "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or device.

[0032] It should be understood that in this invention, "multiple" refers to two or more. "And / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, "and / or B" can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "Contains A, B, and C", "Contains A, B, and C" means that all three A, B, and C are contained; "Contains A, B, or C" means that one of A, B, and C is contained; "Contains A, B, and / or C" means that any one, two, or three of A, B, and C are contained.

[0033] It should be understood that in this invention, "B corresponding to A", "B corresponding to A", "A and B correspond", or "B and A correspond" means that B is associated with A, and B can be determined based on A. Determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information. Matching A and B means that the consistency between A and B is greater than or equal to a preset threshold.

[0034] Depending on the context, "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection."

[0035] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0036] In areas of my country prone to frequent geographical disasters, there are usually established disaster relief command centers for centralized emergency command during disasters. These command centers, in addition to standard office equipment and computer management systems (used to collect, process, analyze, and store disaster information to provide data support for decision-makers), are typically equipped with emergency power systems to ensure normal operation during power outages. They also include audio and video acquisition and transmission equipment such as cameras, microphones, and speakers for real-time disaster monitoring, gathering on-site information, and conducting teleconferences. However, unlike emergency power systems, while command centers also have communication equipment, this equipment still needs to be temporarily supplied by service providers. When basic communication equipment is damaged and cannot be fully repaired in a timely manner, communication service providers will provide temporary communication services using emergency communication vehicles or drones. The broadband resources in these temporary communication services are allocated by the communication service provider, typically using Static Bandwidth Allocation (SBA). This is a common bandwidth allocation method that pre-allocates a fixed amount of bandwidth based on the actual needs and priorities of each service. After a disaster, for the purpose of overall management, telecommunications service providers will prioritize allocating broadband resources to disaster relief command centers. This ensures the bandwidth needs of the command centers, but may also lead to a waste of bandwidth resources. These devices play a crucial role in disaster relief command centers, improving command efficiency, coordinating rescue operations, and ensuring accurate information transmission. In disaster areas, disaster relief command centers are vital hubs for rescue work, and it is essential to ensure the completeness and normal operation of their equipment. For example, for conducting on-site command meetings. Currently, disaster relief command centers typically have strong local hardware reserves, but during disaster relief, the communication bottleneck is often related to lines, rendering the powerful equipment in the command centers often unused. On the other hand, online communication is undoubtedly most effective based on video, followed by voice communication, while text communication is the least efficient. However, the demand for bandwidth resources is exactly the opposite.

[0037] The basic idea of ​​this invention is to configure the terminals connected to the system to conduct calls on demand and according to the scenario, based on application requirements. Simultaneously, during conference calls, there is specific service detection between the terminals and the communication service component, enabling service degradation or upgrades (audio / video reduced to voice or text messages). Furthermore, through a built-in speech recognition and conversion component, the ongoing conference call can be converted into a specific text message, which is then sent to the corresponding terminal via a text messaging service component. This invention adopts the following technical solution: a multi-source fusion communication system for emergency situations, comprising multiple communication terminals, at least one of which includes a command client. The command client includes at least an audio / video acquisition device and an information transmission device. The command client also has a communication service component installed, wherein the audio acquisition device is equipped with a speech recognition and conversion component to convert speech into text information.

[0038] The video capture device is equipped with a streaming media processing module for processing streaming media files. This module includes an identity preset value module, which defines the identities of people appearing in the video. The streaming media processing module splits the video into multiple images and audio information, identifies people in the images and associates them with their corresponding expressions, actions, and voices, and then sends the extracted voice portion to a speech recognition and conversion component for conversion.

[0039] Audio acquisition equipment is required for voice extraction. This equipment is crucial for converting sound signals into electrical signals and includes, but is not limited to, microphones, pickups, audio acquisition cards, embedded recording microphones, and other multi-functional recorders and professional audio equipment. Microphones, as the most basic audio acquisition tool, use the principle of electroacoustic conversion, employing a diaphragm and electromagnetic coil to convert sound waves into electrical signals, and are widely used in the communication system described in this invention. When outdoor communication is required, the command client must at least be equipped with a pickup. A pickup is an audio acquisition device specifically designed for a particular environment or application scenario, typically possessing high sensitivity and anti-interference capabilities. During the editing process, an audio acquisition card is needed to convert analog audio signals into digital signals for easy storage, editing, and transmission.

[0040] In addition, the command client also includes a text information service component, which converts the expressions, actions, and background information recognized in the image into text information, and obtains the text information output by the speech recognition and conversion component, and organizes the text information according to the relationships between the people to form the output text.

[0041] The client is instructed to select the actual output content based on the predicted bandwidth usage.

[0042] Each communication terminal transmits signals carrying GIS location information. The communication terminal also includes a BeiDou satellite short message channel support device, which is connected to a text messaging service component. The BeiDou satellite short message channel support device set in the command client includes at least a BeiDou satellite receiving antenna, a signal amplifier, and a signal modem. These devices are used to transmit and receive signals from the BeiDou satellite, amplify and demodulate the received signals to restore the original short message information, and compress and transmit the transmitted signals. The main function of the BeiDou satellite receiving antenna is to receive and transmit signals from the BeiDou satellite and transmit them to the signal amplifier for easier processing. The signal modem demodulates the amplified signal to restore the original short message information. The audio / video acquisition device includes a morphological recognition conversion component. This component automatically generates descriptive text based on the facial expressions and behaviors of the focus person in the video.

[0043] During operation, the system directs clients to conduct conference calls normally, and captures video and audio information from the conference using audio / video capture devices. The communication service component monitors allocated bandwidth resources in real time during video conferences.

[0044] For network bandwidth monitoring, several mature technologies and solutions exist. Network bandwidth monitoring is a crucial step in ensuring network stability and efficiency, primarily involving real-time monitoring and analysis of network bandwidth usage. SNMP (Simple Network Management Protocol) is a tool used to collect information from network devices and monitor network performance. Through it, administrators can obtain key information such as bandwidth usage and connection status of network devices. Open-source tools such as Nagios and Zabbix support various network devices and services, allowing for customizable monitoring thresholds and alarm mechanisms to achieve real-time monitoring and anomaly alerts for network bandwidth. Professional monitoring software such as SolarWinds NPM (Network Performance Monitor) and Paessler PRTG provide more comprehensive network performance monitoring functions, including bandwidth monitoring, latency monitoring, and packet loss monitoring, and support visualization and data analysis. Real-time traffic monitoring tools such as iftop and nload can display real-time bandwidth usage, helping administrators quickly locate network bottlenecks. Some monitoring tools also support the collection and analysis of historical data. By analyzing historical bandwidth usage data, future bandwidth needs can be predicted, providing a basis for network expansion and optimization.

[0045] Simultaneously, based on the demand for other services, the broadband usage for the next time period is predicted, thereby calculating the remaining broadband resources for the next time period. The information transmission equipment initiates communication service renegotiation and outputs content according to the assessed communication resources. The assessment is performed based on communication resource thresholds, and the collected video and audio information is processed sequentially according to different thresholds as follows:

[0046] Reduce video resolution;

[0047] Reduce overall streaming bitrate;

[0048] Switch to voice access;

[0049] Switch to sending text messages.

[0050] The method of predicting bandwidth usage for the next time period based on the demand of other services employs a cloud computing platform bandwidth monitoring solution when network conditions are good; otherwise, a QoS policy is used. Network bandwidth adjustment technology dynamically allocates and optimizes network bandwidth resources based on monitoring results to improve network performance and user experience. The cloud computing platform's bandwidth monitoring solution enables real-time monitoring and management of cloud resources, ensuring the stability and efficiency of cloud services. QoS policies classify and prioritize network traffic, ensuring critical services receive sufficient bandwidth and low latency, and controlling traffic based on factors such as application type and user identity. Traffic shaping and rate limiting technologies smooth out network traffic bursts, reducing bandwidth waste and congestion, and limiting the rate of specific traffic to prevent it from consuming excessive bandwidth resources. Network topology optimization, through measures such as increasing the number of switches and routers and adopting more efficient transmission protocols, can reduce network latency and bandwidth bottlenecks, improving overall network performance.

[0051] Before using the image recognition feature, the command client needs to manually input the designated commander and upload the corresponding commander's image. If there are multiple commanders, their command order must be specified. The text messaging service component captures the speaker's expression, actions, and the objects of those actions, and converts them into text descriptions. When sending text messages, the text messaging service component includes corresponding text descriptions of the expression and actions.

[0052] To ensure consistent communication, during the meeting, service quality monitoring between the terminal and the communication service component detects a decline in audio and video communication quality. When a threshold is reached, the communication service component predicts the duration of adjusted broadband resources and selects a processing scheme that can stably maintain the connection for a certain period. Before switching to a new scheme, the component provides voice or text prompts. This single-stage maintenance time is no less than 15 minutes to avoid frequent switching that could further impact communication efficiency. In the event of a complete loss of broadband resources, information is transmitted via the short message channel of the BeiDou satellite system.

[0053] like Figure 1 As shown, the specific business process is as follows:

[0054] - It is necessary to conduct emergency command in the disaster area, analyze and convert the video and audio information generated in real time by the command client, and ensure that video, audio and text information are available at the same time. According to the actual communication resource situation, invite communication terminals with good communication resources to join the call conference according to audio and video, such as communication terminal A; invite terminals with scarce communication resources to join the call conference according to voice, such as communication terminal B.

[0055] - During the meeting, service quality monitoring between communication terminal A and the communication service component revealed that the audio and video communication quality was deteriorating. When a threshold was reached, the communication service component initiated a communication service renegotiation with communication terminal A, based on the assessed communication resources.

[0056] Reduce resolution;

[0057] Reduce bitrate;

[0058] Downgraded to voice access;

[0059] Switch to sending text messages.

[0060] The priority is determined as above. When it is determined that the resolution should be reduced, the communication service component and communication terminal A negotiate communication resources that are lower than the current service resolution, and then continue to calculate and determine the service quality.

[0061] If the quality of the business changes, continue to make adjustments following the steps above.

[0062] Voice and text processing: When the service quality between the terminal and the communication service component is so poor that even voice communication cannot be maintained, the communication service component initiates interaction with the speech recognition and conversion component to automatically convert the voice in the current call into text. After recognition, the communication service component sends the corresponding text message to the communication terminal through the text messaging service component.

[0063] After receiving the message, the communication terminal can reply. The reply includes GIS location information, facilitating unified command and management by the command client. At this point, the text messaging service receives the message and forwards it to the command client, notifying the command personnel at the rear. This allows for low-resource-consumption interaction between the two.

[0064] Before switching processing methods, the system provides voice or text prompts indicating the equipment to be replaced and the chosen solution. Even when broadband resources are completely lost, it can promptly switch to transmitting information via the BeiDou satellite short message channel.

Claims

1. A multi-source fusion communication system for emergency situations, characterized in that, It includes multiple communication terminals, at least one of which includes a command client. The command client includes at least an audio / video acquisition device and an information transmission device. The command client also has a communication service component installed, wherein the audio acquisition device is equipped with a speech recognition and conversion component to convert speech into text information. The video capture device is equipped with a streaming media processing module for processing streaming media files. This module includes an identity preset value module, which defines the identities of people appearing in the video. The streaming media processing module splits the video into multiple images and audio information, identifies people in the images and associates them with their corresponding expressions, actions, and voices, and then sends the extracted voice portion to a speech recognition and conversion component for conversion. It also includes a text information service component, which converts the expressions, actions, and background information recognized in the image into text information, and obtains the text information output by the speech recognition and conversion component, and organizes the text information according to the relationships between the people to form the output text. The client is instructed to select the actual output content based on the predicted bandwidth usage. Each communication terminal transmits signals carrying GIS location information. The command client conducts normal telephone conferences, while audio / video acquisition devices capture video and audio information from the conference. The communication service component monitors allocated broadband resources in real-time during the video conference and predicts broadband usage for the next time period based on the needs of other services, thereby calculating the remaining broadband resources for the next time period. The information transmission device initiates communication service renegotiation and outputs content based on the assessed communication resources. This assessment is performed based on communication resource thresholds, and the collected video and audio information is processed sequentially according to different thresholds: Reduce video resolution; Reduce overall streaming bitrate; Switch to voice access; Switch to sending text messages.

2. A multi-source fusion communication system for emergency situations as described in claim 1, characterized in that, The communication terminal also includes a short message channel support device for the BeiDou satellite. The support device is connected to the text information service component. The short message channel support device for the BeiDou satellite set in the command client includes at least a BeiDou satellite receiving antenna, a signal amplifier, and a signal modem. Other communication terminals only include a BeiDou satellite receiver.

3. A multi-source fusion communication system for emergency situations as described in claim 2, characterized in that, The other communication terminals include individual soldier equipment and vehicle-mounted cluster terminals.

4. A multi-source fusion communication system for emergency situations as described in claim 1, characterized in that, The method of predicting bandwidth usage for the next time period based on the demand of other services adopts a bandwidth monitoring solution of the cloud computing platform when the network conditions are good, and otherwise adopts a QoS policy.

5. A multi-source fusion communication system for emergency situations as described in claim 1, characterized in that, Before using the image recognition client to identify people, you need to manually input the designated commander and upload the corresponding commander's image. If there are multiple commanders, you need to specify their command order.

Images