High-reliability digital broadcasting system and abnormal fast continuous broadcasting method

Abstract

The invention discloses a high-reliability digital broadcasting system and an abnormal fast continuous broadcasting method, and relates to the technical field of communication technology and emergency broadcasting, and the high-reliability digital broadcasting system comprises a server A, a server B, broadcasting terminal equipment and a server abnormity recovery module, the server A is provided with two redis sentry nodes, and the server B is provided with one redis sentry node; the server exception recovery module is used for switching the broadcast and quickly continuing the broadcast when the server is exceptional; when a server is started, a redis monitoring sub-process is started, and the redis monitoring sub-process performs fault transfer and master and slave node adjustment according to a local state when a master node is offline by checking the local state, the master node state and redis sentinel node information. According to the invention, when the server is abnormal, the broadcast task which is being executed can be continuously executed within one second, so that the continuous playing efficiency is greatly improved; according to the method, the redis sentry cluster is deployed by using the two servers, so that the number of the servers can be reduced, and the cost is effectively reduced.

Description

A highly reliable digital broadcasting system and a method for exceptionally fast resume playback. Technical Field

[0001] This invention relates to the field of network communication technology, and more specifically, to a highly reliable digital broadcasting system and a method for exceptionally fast resume broadcasting. Background Technology

[0002] A typical digital broadcasting system includes a main control server, broadcast terminal equipment, and digital IP microphone / encoding terminal equipment. The main control server is the key component of the system, primarily responsible for task scheduling, media library storage, media stream transmission, and media stream forwarding from the digital IP microphones and encoding terminal equipment. Prolonged operation of the main control server may lead to hardware or operating system failures. When such problems occur, the entire broadcasting system will become unusable, requiring maintenance personnel to handle the situation and typically taking a considerable amount of time to restore normal operation.

[0003] A typical solution involves deploying two servers, a primary and a backup. A virtual IP address is created using the VRRP protocol, and broadcast terminal devices and microphone / encoding terminal devices are configured to use this virtual IP address. When the primary server is functioning correctly, the virtual IP address maps to the primary server, which then provides services. When the primary server fails, the virtual IP address maps to the backup server, which then provides services. This solution has the following drawbacks:

[0004] 1) Server fault detection and virtual IP address switching typically take a few seconds;

[0005] 2) The function of switching to the backup server when the primary server fails and continuing the execution of broadcast tasks that are currently being executed has not been implemented.

[0006] Therefore, there is an urgent need for a highly reliable digital broadcasting system and an exceptionally fast replay method to solve the above problems. Summary of the Invention

[0007] To address the aforementioned technical problems in related technologies, this invention proposes a highly reliable digital broadcasting system and a method for extremely fast resume broadcasting.

[0008] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0009] In a first aspect, the present invention provides a highly reliable digital broadcasting system, including two servers, broadcasting terminal equipment and server anomaly recovery module;

[0010] The two servers include server A and server B; the two servers share the same virtual IP address; the broadcast terminal device includes an audio input device and a broadcast terminal device; the broadcast terminal device is connected to both servers respectively; the servers include device shadow, streaming service and relay service;

[0011] The two servers maintain heartbeat communication with a first preset time interval. During heartbeat communication, the server sends its own server status to the other server. If a server in standby mode does not receive a heartbeat message from the other server for more than a second preset time, it sets its own server status to active. When the server status changes, the latest status is immediately sent to the broadcast terminal device. The broadcast terminal device synchronizes data with the device shadow in the server.

[0012] Both servers contain Redis, and Redis service and Redis Sentinel service are deployed on each server respectively; two Redis Sentinel nodes are deployed on server A, and one Redis Sentinel node is deployed on server B.

[0013] The server failure recovery module is used to switch to fast resume broadcasting in the event of a server failure; specifically, it includes:

[0014] When the server starts, it initiates a Redis monitoring subprocess and then continuously checks the network interface card (NIC) status. The NIC status includes connected and disconnected states. When the NIC status changes from disconnected to connected, the server initiates the Redis monitoring subprocess and continues to check the NIC status. When the NIC status changes from connected to disconnected, the server sets itself to standby mode, sends a command to stop the Redis monitoring subprocess, and then continues to check the NIC status.

[0015] The Redis monitoring subprocess checks the local machine status, master node status, and Redis sentinel node information, and performs failover and master-slave node adjustment based on the local machine status when the master node is offline.

[0016] Specifically, the Redis monitoring sub-process includes the following steps:

[0017] S11. Check if the server status of this machine is in standby mode. If yes, stop the local Redis service and then proceed to step S12; otherwise, proceed directly to step S12.

[0018] S12. Check if a command to stop the Redis monitoring subprocess has been received. If yes, stop the Redis monitoring subprocess and enter the stopped state; otherwise, proceed to step S13.

[0019] S13. Obtain the master node IP, master node status, slave node IP, and Redis Sentinel node information;

[0020] S14. Check if the master node is offline. If yes, proceed to step S15; otherwise, proceed to step S111.

[0021] S15. Check if the server status of this machine is working. If yes, proceed to step S16; otherwise, proceed to step S13.

[0022] S16. Try pinging the local Redis node. If the ping fails, start the local Redis service and then proceed to step S17. If the ping succeeds, proceed directly to step S17.

[0023] S17. Notify the local sentinel node to perform a failover: sentinel failover;

[0024] S18. Check if the failover was successful. If yes, proceed to step S12; otherwise, proceed to step S19.

[0025] S19. Notify the local Redis node that it will no longer act as a slave node: slaveof no one;

[0026] S110. Check if only one sentinel node is deployed on the local machine. If so, modify the local Redis sentinel service configuration, set the local Redis service as the master node, set the Redis service on the other server as the slave node, and restart the Redis sentinel service. Proceed to step S12. If not, proceed directly to step S14.

[0027] S111. Check if the number of online sentinels is 3 and pinging the local Redis node fails. If yes, start the local Redis service; otherwise, proceed to step S12.

[0028] Specifically, the server anomaly recovery module also includes:

[0029] If server A has a Redis service as the master node and server B has a Redis service as the slave node; if server A is disconnected from the network cable or shut down:

[0030] Server A's synchronization service detects that the network card status has become disconnected, so the server sets its local machine to standby mode and sends a command to stop the Redis monitoring subprocess, thus stopping the Redis monitoring subprocess.

[0031] Server B detects that the master node is offline, and that its own server status is active. If a ping to its local Redis instance is successful, it then executes the following commands sequentially: `sentinel failover` to notify its local sentinel node to perform a failover; `slaveof no one` to notify its local Redis service to no longer act as a slave node; it modifies the Redis sentinel node configuration on server B, setting server B's Redis service as the master node and server A's Redis service as the slave node; and finally restarts server B's Redis sentinel service, thus achieving master node failover.

[0032] When server A reconnects the network cable or powers on, server A checks the network card status and changes it from disconnected to connected. Server A then starts the Redis monitoring subprocess. Since server A's server status is in standby mode, the local Redis service is stopped.

[0033] When server A detects that the master node is online and has been recognized by 3 sentinel nodes, it starts the Redis service on server A. At this time, the sentinels will set the Redis service on server A as a slave node.

[0034] Specifically, the server anomaly recovery module also includes:

[0035] If server B has a Redis service as the master node and server A has a Redis service as the slave node, and server B is disconnected from the network cable or shut down:

[0036] Server B checks that the network card status has changed to disconnected. The server sets its local machine to standby mode and sends a command to stop the Redis monitoring subprocess, thus stopping the Redis monitoring subprocess.

[0037] Server A detects that the master node is offline, checks that its own server status is active, and successfully pings its own Redis. It then notifies its local sentinel nodes to perform a failover. Since server A still has two sentinel nodes online, which is more than half, it can successfully elect its own Redis node as the master node.

[0038] When server B reconnects the network cable or powers on, server B checks the network card status and changes it from disconnected to connected. Server B then starts the Redis monitoring subprocess. Since the local server is in standby mode, the local Redis service is stopped.

[0039] When server B detects that the master node is online and has been recognized by 3 sentinel nodes, it starts the local Redis service. At this time, the sentinels will set server B's Redis service as a slave node.

[0040] Specifically, the Redis monitoring subprocess is a thread.

[0041] Specifically, the two servers use the VRRP protocol to create the same virtual IP address, which is also used for client-side web management access.

[0042] Specifically, the broadcast system maintains a device shadow for each broadcast terminal device through a server, and the broadcast terminal device synchronizes data with the device shadow in the server; the device shadow contains a broadcast module; the broadcast terminal device executes broadcast tasks according to the broadcast control parameters in the broadcast module.

[0043] Specifically, the system also includes a database synchronization module for bidirectional data synchronization between the databases on the two servers; the databases on the two servers are MySQL databases, and two MySQL master-slave synchronization operations are started in the MySQL database services on the two servers to achieve bidirectional data synchronization.

[0044] Specifically, the system also includes a media library synchronization module for bidirectional file synchronization between the media libraries on the two servers; the bidirectional file synchronization includes synchronization of newly added files and synchronization of deleted files.

[0045] Specifically, the system also includes a real-time stream synchronization module for broadcasting real-time stream synchronization between two servers.

[0046] Specifically, the broadcast terminal device periodically queries the status of the two servers according to a third duration, and only accepts control messages sent by servers whose status is working; if the server status changes, the server will immediately and proactively send server status information to the terminal device.

[0047] Secondly, the present invention provides a method for rapid resuming of high-reliability digital broadcasting in case of anomalies, based on the high-reliability digital broadcasting system described in the first aspect, characterized by comprising the following steps:

[0048] S1. When server A loses power, the server fault recovery module sets server B as the master node; where server A is the master node and server B is the slave node.

[0049] S2. The broadcast terminal equipment receives a message from the working server that it has changed from server A to server B.

[0050] S3. The broadcast terminal equipment synchronizes the device shadow from server B, where the broadcast module data is consistent with that before the switch.

[0051] S4. The broadcast terminal device requests server B to relay the real-time stream. The request parameters are the stream address and stream ID in the broadcast module.

[0052] S5. After receiving the request, the relay service of server B pulls the RTMP stream from the stream service.

[0053] S6. Server B relay service will convert the real-time stream into an RTP stream and send it to the broadcast terminal.

[0054] S7. After receiving the RTP stream, the broadcast terminal equipment decodes and plays it.

[0055] This invention provides a highly reliable digital broadcasting system and a method for rapid resume broadcasting in case of an anomaly. When a server malfunctions, the broadcasting task being executed can be resumed within 1 second, greatly improving the efficiency of resume broadcasting.

[0056] Furthermore, the present invention uses two servers to deploy a Redis Sentinel cluster, which can reduce the number of servers and effectively reduce costs. When using three or more servers to deploy a Redis Sentinel cluster, the system can only support less than half of the server failures during normal operation. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 is a schematic diagram of a high-reliability digital broadcasting system according to an embodiment of the present invention;

[0059] Figure 2 is a schematic diagram of server MySQL master-slave synchronization provided according to an embodiment of the present invention;

[0060] Figure 3 is a schematic diagram of the file synchronization process provided according to an embodiment of the present invention;

[0061] Figure 4 is a schematic diagram of the real-time broadcast stream synchronization process provided according to an embodiment of the present invention;

[0062] Figure 5 is a schematic diagram of the broadcasting process of a broadcasting terminal device according to an embodiment of the present invention;

[0063] Figure 6 is a schematic diagram of the deployment of a Redis server and a Redis sentinel node according to an embodiment of the present invention;

[0064] Figure 7 is a schematic diagram of a server anomaly recovery module provided according to an embodiment of the present invention;

[0065] Figure 8 is a schematic diagram of a Redis monitoring sub-process provided according to an embodiment of the present invention;

[0066] Figure 9 is a schematic flowchart of a method for rapid resuming of high-reliability digital broadcasting in case of anomalies, provided by an embodiment of the present invention. Detailed Implementation

[0067] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0068] In the description of this invention, the use of terms such as "first," "second," etc., is for the purpose of distinguishing technical features only and should not be construed as indicating or implying relative importance, or implicitly indicating the number of technical features indicated, or implicitly indicating the order of the technical features indicated.

[0069] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, etc., are based on the orientation or positional relationship shown in the drawings and are only for the convenience of describing this invention and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0070] In the description of this invention, it should be noted that, unless otherwise explicitly defined, terms such as "setting," "installation," and "connection" should be interpreted broadly. Those skilled in the art can reasonably determine the specific meaning of these terms in this invention based on the specific content of the technical solution.

[0071] Example 1.

[0072] Referring to Figure 1, this embodiment provides a highly reliable digital broadcasting system and a method for rapid resume playback in case of anomalies, including two servers, broadcast terminal equipment, a database synchronization module, a media library synchronization module, a real-time stream synchronization module, and a server anomaly recovery module; the two servers include server A and server B; the broadcast terminal equipment includes an audio input device and a broadcast terminal device; the broadcast terminal equipment is connected to the two servers respectively; the servers include device shadows, streaming services, and relay services;

[0073] The two servers use the VRRP protocol to create the same virtual IP address, as shown in Figure 1. The virtual IP address is also used for client web management access.

[0074] VRRP (Virtual Router Redemption Protocol) allows multiple routers (or servers, firewalls, and other gateway devices) to share a single virtual IP address. When one device fails, another device can seamlessly take over the virtual IP, thus ensuring the continuity of network services. This is existing technology and will not be elaborated upon here.

[0075] System users access the broadcast system management page on their client machines using a browser, entering the virtual IP address in the browser's address bar to access the system.

[0076] If the server mapping the virtual IP address fails, it can automatically map to another server, allowing users to continue using the broadcast system without having to reopen the webpage or switch the system page address.

[0077] The server states of the two servers include: working state, standby state, and offline state; the server states of the two servers are mutually exclusive, that is, only one server is in the working state, and the other server is in the standby or offline state.

[0078] For example, if server A is in a working state, then server B is in a standby or offline state.

[0079] The two servers maintain heartbeat communication with a first preset time interval. During heartbeat communication, the server sends its own server status to the other party. When a server is in standby mode and does not receive a heartbeat message from the other server for more than a second preset time (heartbeat timeout), it sets its own server status to working mode and sets the other server to offline mode. When the server status changes, the latest server status is immediately sent to the broadcast terminal device.

[0080] Both servers include a database and a media library. The media library is responsible for storing actual media files (videos, audios, images, and documents), while the database is responsible for managing metadata and business logic.

[0081] The database synchronization module is used for bidirectional data synchronization between databases on two servers.

[0082] Referring to Figure 2, in this embodiment, the databases of the two servers are MySQL databases. Two MySQL master-slave synchronization operations are started in the MySQL database services on the two servers to achieve bidirectional data synchronization.

[0083] MySQL master-slave replication, also known as MySQL Replication, enables data to be synchronized from one database server to multiple database servers. MySQL databases have built-in master-slave replication functionality, and with configuration, various master-slave replication schemes based on database and table structures can be implemented. This is existing technology and will not be elaborated upon here.

[0084] The media library synchronization module is used for bidirectional file synchronization between media libraries on two servers; the bidirectional file synchronization includes synchronization of newly added files and synchronization of deleted files.

[0085] Referring to Figure 3, the process for synchronizing new files is as follows:

[0086] After a file is uploaded to the media library, a record is created in the database, while the media file is stored in the file storage service (minio).

[0087] a1) Enable the event mechanism of the minio service. After the minio service uploads a file, it sends the message of the newly added file to the message queue (RabbitMQ).

[0088] b) The synchronization service on server B subscribed to the message queue of server A and received the new file message;

[0089] c) The synchronization service on server B downloads newly added files from server A (before downloading, it checks whether the file already exists in the local minio and compares the file digest (MD5) to see if they match. If the file already exists, the process stops to avoid an infinite loop).

[0090] d) The synchronization service on server B uploads the new file to the local minio.

[0091] The process for synchronizing deleted files is similar to that for synchronizing new files, as detailed below:

[0092] When a file is deleted from the media library, a record is created in the database, while the media file itself is stored in the file storage service (minio).

[0093] e) Enable the event mechanism of the minio service. After deleting a file in server A, the minio service will send the file deletion message to the message queue (RabbitMQ).

[0094] f) The synchronization service on server B subscribed to the message queue of server A and received the file deletion message;

[0095] g) The synchronization service in server B checks whether the file exists in the local minio file. If it exists, it performs a deletion operation.

[0096] h) The synchronization service on server B deletes the corresponding file from the local minio.

[0097] The real-time stream synchronization module is used for broadcasting real-time stream synchronization between two servers;

[0098] Referring to Figure 4, the specific process of real-time broadcast stream synchronization is as follows:

[0099] After the broadcast task is started on server A, the broadcast application service calls the push streaming service to start file push streaming.

[0100] The push streaming service in server A downloads files from the local minio and pushes them to the local streaming service. Assume the streaming address is: rtmp: / / 127.0.0.1 / live / 000001;

[0101] The streaming service in server A forwards the streaming request to the streaming service in server B, and the request parameters are the same as those received by the broadcast application service.

[0102] The streaming service on server B downloads the same file from its local miniio repository and pushes it to its local streaming service, using the same streaming address as on server A. (The miniio repository on server B has already synchronized the media files from server A.)

[0103] A streaming service is a server-side program that receives, processes, and distributes real-time audio and video streams, typically based on streaming media protocols such as RTMP, HLS, and WebRTC. It is responsible for forwarding media data uploaded by the streaming service to clients (broadcast terminal devices), supporting simultaneous streaming and viewing by multiple users.

[0104] Broadcast terminal equipment has two methods for streaming:

[0105] a) When deploying a wide area network, terminal devices directly pull streams from the streaming service, typically using the RTMP protocol.

[0106] b) In a local area network (LAN) deployment: The relay service pulls the audio stream from the streaming service (RTMP protocol), and then the relay service uses the RTP protocol to send the audio stream to the terminal devices. This method helps improve the synchronization of audio playback on terminal devices.

[0107] A streaming service is middleware responsible for acquiring media data from the source (audio input device) and pushing it to the streaming service via streaming media protocols. It typically includes functions such as file reading, encoding, and network transmission.

[0108] The broadcast terminal device simultaneously queries the status of the two servers at regular intervals according to the third duration, and only accepts control messages sent by the server whose status is working; if the server status changes, the server will immediately and proactively send server status information to the terminal device.

[0109] The audio input device includes a digital IP microphone or an encoding terminal device;

[0110] The first duration is 100ms; the second duration is 500ms; and the third duration is 1000ms.

[0111] Similar to the Internet of Things (IoT) system, the broadcast system in this embodiment maintains a device shadow for each broadcast terminal device through a server, and the broadcast terminal device synchronizes data with the device shadow in the server;

[0112] The device shadow includes a broadcast module, which contains broadcast control parameters such as broadcast switch, task ID, stream address, volume, and name. The broadcast terminal device executes the broadcast task according to the broadcast control parameters in the broadcast module.

[0113] Referring to Figure 5, this embodiment also provides a broadcasting process for a broadcasting terminal device, specifically including:

[0114] Broadcast terminal equipment synchronizes with the broadcast module from the device shadow.

[0115] When the broadcast switch in the broadcast module is set to "Enable Broadcast", the broadcast terminal device requests the relay service to forward the real-time stream, and the request parameters are the stream address and stream ID in the broadcast module.

[0116] After receiving a request, the relay service pulls the real-time stream (RTMP or RTSP stream) from the stream service.

[0117] The relay service converts real-time streams (RTMP or RTSP streams) into RTP streams and sends them to broadcast terminal devices.

[0118] After receiving the RTP stream, the broadcast terminal equipment decodes and plays it.

[0119] When deployed on a local area network, terminal devices use a relay service to proxy and pull streams, converting the RTMP protocol to the RTP protocol to improve the synchronization of playback across multiple devices.

[0120] In a broadcast task, multiple devices may request the relay service to forward the same real-time stream. The relay service only needs to pull one stream from the stream service and then send it to the terminal device via multicast or unicast using the RTP protocol.

[0121] This system includes two types of broadcast tasks: file broadcast tasks and real-time broadcast tasks;

[0122] The file broadcasting task is to push media files from the media library to the broadcast terminal device through a working server;

[0123] When the system starts a file broadcast task, both servers simultaneously start pushing media files to the local streaming media service; the server in working state sends broadcast control instructions to the broadcast terminal device; after receiving the broadcast control instructions from the server in working state, the broadcast terminal device pulls the media stream from its streaming media service and plays it.

[0124] If a working server fails during the execution of a file broadcast task, another server will immediately switch to working status and notify all terminal devices. After the terminal devices are notified of the server switch, they will immediately obtain the current broadcast task from the new working server and continue to execute the current broadcast task.

[0125] Real-time broadcasting tasks involve digital IP microphones or encoding terminal devices capturing audio from microphone or line inputs, encoding it into digital audio, and then pushing it to a server, which then forwards it to broadcast terminal devices for broadcasting.

[0126] Once the real-time broadcast task begins, the digital IP microphone or encoding terminal device pushes the media stream to the working server, which then forwards it to the broadcast terminal device for broadcast.

[0127] During the execution of a real-time broadcast task, if a server in operation fails, another server immediately switches to operation and notifies all terminal devices. Upon notifying the switch, digital IP microphones or encoding terminal devices immediately push their media streams to the new operating server. Upon notifying the switch, broadcast terminal devices immediately retrieve the current broadcast task from the new operating server and continue executing it.

[0128] Referring to Figure 6, in this embodiment, both servers contain a Redis instance, and Redis service and Redis Sentinel service are deployed respectively.

[0129] Redis is an open-source, high-performance key-value store database that provides various data structures for storing data, such as strings, hashes, lists, sets, and sorted sets. Redis stores data in memory for fast read and write access and can asynchronously persist data to disk.

[0130] The device shadow data is stored in the server's Redis cache. Two servers each deploy a Redis service, and the other two servers deploy three Redis sentinel nodes to achieve Redis data synchronization between the two servers.

[0131] It is worth noting that in existing technologies, a Redis Sentinel cluster typically uses at least three servers, with one Redis Sentinel node deployed on each server. This is because when the Redis master node fails, the Sentinel nodes use an election mechanism to elect a new Redis master node, which requires more than half of the Sentinel votes to complete the election.

[0132] This embodiment innovatively uses two servers to deploy a Redis Sentinel cluster. Server A has two Redis Sentinel nodes: the first Sentinel node and the second Sentinel node, while server B has one Redis Sentinel node: the third Sentinel node. When a server fails, the server can be quickly switched over through the synchronization service and the server failure recovery module, and the broadcast task being executed can continue to be executed within 1 second.

[0133] A Redis service is a Redis node used for caching data read and write. Server A and server B each deploy a Redis service, and under normal circumstances, one Redis service is the Redis master node, or master node, and the other Redis service is the Redis slave node, or slave node. The master node can provide read and write caching to the outside world, while the slave node only provides read caching to the outside world. The slave node automatically synchronizes the cached data of the master node.

[0134] After the initial installation, the Redis service on server A is the master node, and the Redis service on server B is the slave node; subsequently, the status of the Redis service is determined by the Redis Sentinel service through an election mechanism if the master node fails.

[0135] The Redis service has the following states: online, offline, and without a elected master node. In this embodiment, both offline and without a elected master node are considered as offline states.

[0136] The Redis Sentinel node is used to create a Redis Sentinel cluster. It does not store cached data itself. Its main purpose is to have one master node (Redis service) in the cluster, and other Redis services as slave nodes (to synchronize cached data from the master node). The Redis Sentinel service is a collection of functions provided by the Sentinel node, including monitoring, notification, and automatic failover.

[0137] In Sentinel mode, a Sentinel cluster is established, and a Sentinel node is connected to the local machine for synchronization services. You can check how many Sentinel nodes have joined this cluster.

[0138] In this embodiment, after server A loses power, less than half of the sentinel nodes remain. Therefore, the synchronization service needs to intervene to set server B's Redis service as the master node.

[0139] In this embodiment, a synchronization service is deployed on each server. The synchronization service is a process that includes the following functions:

[0140] 1) Continuously check the network card status of the server hosting the synchronization service;

[0141] When a server shuts down abnormally, another server uses a heartbeat timeout mechanism to determine the network card status of the server.

[0142] 2) Assist in the fault recovery of the Redis Sentinel cluster, mainly solving the problem of abnormal recovery of the Redis master node;

[0143] 3) Switch the server status;

[0144] 4) Obtain the status of the master node from the Redis Sentinel service on the local machine;

[0145] Because the local sentinel periodically sends heartbeat (PING) and cluster topology query (INFO) commands to the master node and negotiates with other sentinels to confirm the master node's status, thereby obtaining and maintaining the latest information of the master node in real time, the status of the master node can be obtained from the local Redis sentinel service;

[0146] When the Redis service on server A is the master node, the Redis service on server B is the slave node. After server A loses power, only one sentinel node remains on server B. Therefore, by default, the Redis service on server B cannot be elected as the master node.

[0147] When server A goes offline abnormally, server B's synchronization service immediately notifies server B's sentinel node to perform a failover: "sentinel failover". If the failover fails, the following operations are performed: control the Redis service to exit the slave node state: "slaveof no one", then modify the configuration file of the third sentinel node and restart the Redis sentinel service, and reset the Redis master node. Through the above mechanism, the device shadow data in the broadcast system is synchronized in real time between the two servers.

[0148] Sentinel failover is a failover command;

[0149] `slaveof no one` is a Redis command used to immediately promote a slave node to master. After execution, the instance will disconnect from the original master node's replication connection, handle read and write requests independently, and retain the replicated data.

[0150] It is worth noting that the server is in standby mode by default after a restart; the Sentinel Node service starts automatically upon startup; and the Sentinel Nodes in the Sentinel Cluster automatically communicate with each other to determine the master node.

[0151] The server failure recovery module is used to switch broadcasts and quickly resume playback when the server fails.

[0152] Referring to Figure 7, which is a schematic diagram of the server anomaly recovery module, it specifically includes:

[0153] When the server starts, it initiates a Redis monitoring subprocess, and then continuously checks the network interface card (NIC) status through a synchronization service; the NIC status includes both connected and disconnected status.

[0154] The Redis monitoring subprocess checks the local machine status, master node status, and Redis sentinel node information, and performs failover and master-slave node adjustment based on the local machine status when the master node is offline.

[0155] In this embodiment, the Redis monitoring subprocess is a single thread that can receive a command to stop the Redis monitoring subprocess and thus be stopped.

[0156] When the network interface card (NIC) status changes from disconnected to connected, the server starts the Redis monitoring subprocess and then continues to check the NIC status.

[0157] When the network interface card (NIC) status changes from connected to disconnected, the server sets its local machine to standby mode, then sends a command to stop the Redis monitoring subprocess, and then continues to check the NIC status. For example, if server A shuts down, server B detects a heartbeat timeout and sets its local machine to working mode; while server A restarts and goes through the boot process.

[0158] If the network card status remains unchanged, continue checking the network card status.

[0159] Referring to Figure 8, the Redis monitoring sub-process specifically includes the following steps:

[0160] S11. Check if the server status of this machine is in standby mode. If yes, stop the local Redis service and then proceed to step S12; otherwise, proceed directly to step S12.

[0161] It is understandable that "this machine" refers to the local server.

[0162] S12. Check if a command to stop the Redis monitoring subprocess has been received. If yes, stop the Redis monitoring subprocess (stop the thread where the Redis monitoring subprocess is located) and enter the stopped state; otherwise, proceed to step S13.

[0163] S13. Obtain the master node IP, master node status, slave node IP, and Redis Sentinel node information;

[0164] Redis Sentinel node information refers to the parameters stored in the Redis Sentinel node configuration file, including: master node address (IP and port) and slave node address (IP and port); the Sentinel node is read from the Redis Sentinel node configuration file when it starts up.

[0165] If the master and slave nodes change during the operation of the Sentinel service, the Sentinel service will automatically update the relevant parameters.

[0166] After modifying the following parameters, restart the Sentinel service for synchronization:

[0167] 1) Master node address, for example: 192.168.1.100:6379:

[0168] sentinel monitor mymaster 192.168.1.100:6379;

[0169] From the node address, for example: 192.168.1.101:6379:

[0170] sentinel known-replica mymaster 192.168.1.101 6379.

[0171] S14. Check if the master node is offline. If yes, proceed to step S15; otherwise, proceed to step S111.

[0172] S15. Check if the server status of this machine is working. If yes, proceed to step S16; otherwise, proceed to step S13.

[0173] S16. Try pinging the local Redis node. If the ping fails, start the local Redis service and then proceed to step S17. If the ping succeeds, proceed directly to step S17.

[0174] S17. Notify the local sentinel node to perform a failover: sentinel failover;

[0175] S18. Check if the failover was successful. If yes, proceed to step S12; otherwise, proceed to step S19.

[0176] S19. Notify the local Redis node that it will no longer act as a slave node: slaveof no one;

[0177] S110. Check if only one sentinel node is deployed on the local machine. If so, modify the local Redis sentinel service configuration, set the local Redis service as the master node, set the Redis service on the other server as the slave node, and restart the Redis sentinel service. Proceed to step S12. If not, proceed directly to step S14.

[0178] S111. Check if the number of online sentinels is 3 and pinging the local Redis node fails. If yes, start the local Redis service; otherwise, proceed to step S12.

[0179] This embodiment provides a possible specific implementation of a server anomaly recovery module, as follows:

[0180] If server A has a Redis service as the master node and server B has a Redis service as the slave node; if server A is disconnected from the network cable or shut down:

[0181] Server A's synchronization service detects that the network card status has become disconnected, so the server sets its local machine to standby mode and sends a command to stop the Redis monitoring subprocess, thus stopping the Redis monitoring subprocess.

[0182] If the server shuts down normally, the synchronization service, Redis nodes, and Redis sentinel nodes will automatically stop; the Redis monitoring subprocess will also stop executing during the synchronization service shutdown process.

[0183] Server B detects that the Redis master node is offline, but detects that the local server (Server B) is in a working state. If the local Redis is successfully pinged, it will then execute the following steps: notify the local sentinel node (the third sentinel node) to perform failover: `sentinel failover`, notify the local Redis service to no longer act as a slave node: `slaveof no one`, modify the Redis sentinel node configuration of Server B to set the Redis service of Server B as the master node and the Redis service of Server A as the slave node, and restart the Redis sentinel service of Server B to achieve master node failover.

[0184] When server A reconnects the network cable or powers on, server A checks the network card status and changes it from disconnected to connected. Server A then starts the Redis monitoring subprocess. Since server A's server status is in standby mode, the local Redis service is stopped.

[0185] The sentinel nodes of both server A and server B will retain the reids service information of server A and server B.

[0186] After server A is reconnected to the network cable or restarted, only server B's Redis service is online. Server A's first and second sentinel nodes can only vote for the online server B's Redis service. In addition, server B's third sentinel node can only vote for the online server B's Redis service. The three sentinel nodes will eventually reach a consensus and set server B's Redis service as the master node. At this point, it is considered that the master node has been detected as online and has been recognized by three sentinel nodes.

[0187] When server B detects that the master node (server B's Redis service) is online and has been recognized by 3 sentinel nodes, it starts server A's Redis service. At this time, the sentinels will set server A's Redis service as a slave node.

[0188] This embodiment also provides another possible specific implementation of the server anomaly recovery module, as follows:

[0189] If server B has a Redis service as the master node and server A has a Redis service as the slave node, and server B is disconnected from the network cable or shut down:

[0190] Server B checks that the network card status has changed to disconnected. The server sets its local machine to standby mode and sends a command to stop the Redis monitoring subprocess, thus stopping the Redis monitoring subprocess.

[0191] Server A detects that the master node (Server B's Redis service) is offline, checks that its own server status is working, and successfully pings its own Redis. It then notifies its local sentinel node to perform a failover: `sentinelfailover`. Since Server A still has two sentinel nodes (the first and second sentinel nodes) online, which is more than half, it can successfully elect its own Redis node as the master node.

[0192] When server B reconnects the network cable or powers on, server B checks the network card status and changes it from disconnected to connected. Server B then starts the Redis monitoring subprocess. Since the local server is in standby mode, the local Redis service is stopped.

[0193] The sentinel nodes of both server A and server B will retain the reids service information of server A and server B.

[0194] After server B is reconnected to the network cable or restarted, only the Redis service of server A is online. The first and second sentinel nodes of server A have already voted for the online Redis service of server A, and the third sentinel node of server B will also vote for the online Redis service of server A. The three sentinel nodes will eventually reach a consensus and set the Redis service of server A as the master node. At this point, it is considered that the master node has been detected as online and has been recognized by three sentinel nodes.

[0195] When server A detects that the master node (server A's Redis service) is online and has been recognized by 3 sentinel nodes, it starts the local Redis service. At this time, the sentinel will set the local (server B) Redis service as a slave node.

[0196] In the above process, after the server recovers from the abnormal state (by reconnecting the network cable or restarting), the server's Redis service will be added to the Redis Sentinel cluster for automatic monitoring as a slave node, and the Redis slave node will automatically synchronize all data with the Redis master node.

[0197] Understandably, after deploying a Redis Sentinel cluster on the server, applications access Redis using a common method: the application establishes a connection with one or more Redis Sentinels, and queries the current Redis master and slave nodes through the Sentinels. When writing data, it connects to the Redis master node; when reading data, it can select any node. These operations are already encapsulated in the Redis SDK and do not need to be implemented manually; this is existing technology and will not be elaborated upon here.

[0198] Referring to Figure 9, this embodiment provides a method for rapid resume broadcasting in high-reliability digital broadcasting, including the following steps:

[0199] S1. When server A loses power, the server fault recovery module sets server B as the master node; where server A is the master node and server B is the slave node.

[0200] S2. The broadcast terminal equipment receives a message from the working server that it has changed from server A to server B.

[0201] S3. The broadcast terminal equipment synchronizes the device shadow from server B, where the broadcast module data is consistent with that before the switch.

[0202] S4. The broadcast terminal device requests server B to relay the real-time stream. The request parameters are the stream address and stream ID in the broadcast module.

[0203] S5. After receiving the request, the relay service of server B pulls the RTMP stream from the stream service.

[0204] S6. Server B relay service will convert the real-time stream into an RTP stream and send it to the broadcast terminal.

[0205] S7. After receiving the RTP stream, the broadcast terminal equipment decodes and plays it.

[0206] The duration of the entire switching process depends on the time it takes for server B to switch to working status after server A loses power. As mentioned above, the heartbeat timeout between the two servers is 500 milliseconds. After server A loses power, the broadcast terminal switches to server B after 500 milliseconds and starts requesting to pull the stream and broadcast. The broadcast task being executed can continue to be executed within 1 second, which greatly improves the efficiency of resuming broadcast.

[0207] Alternatively, after receiving a synchronous push request via the push streaming service, the push is delayed by 1 second. During this time, the real-time stream on server B is 1 second slower than that on server A. When server A malfunctions, the terminal switches to server B and resumes broadcasting. During the switching process, the terminal pauses playback for less than 1 second, ensuring that the broadcast content is not lost when the terminal device resumes playback.

[0208] This embodiment uses two servers to deploy a Redis Sentinel cluster, which reduces the number of servers and effectively lowers costs. If three or more servers are used to deploy a Redis Sentinel cluster, the system can only support less than half of the servers failing during normal operation.

[0209] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as a microprocessor, such as a central processing unit, a digital signal processor, or software executed by a microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0210] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A high-reliability digital broadcasting system, characterized in that, The system includes two servers, a broadcast terminal device, and a server anomaly recovery module. The two servers are Server A and Server B, both sharing the same virtual IP address. The broadcast terminal device includes an audio input device and a broadcast terminal device, each connected to one of the two servers. Each server includes a device shadow, a streaming service, and a relay service. The two servers maintain heartbeat communication at a first preset interval, sending their own server status to the other during heartbeat communication. If a server remains in standby mode for more than a second preset time without receiving a heartbeat message from the other server, it sets its own server status to active. When a server's status changes, it immediately sends the latest status to the broadcast terminal device. The broadcast terminal device synchronizes data with the device shadow in the server. Both servers contain Redis, and a Redis service and a Redis sentinel service are deployed on each server. Two Redis sentinel nodes are deployed on Server A, and one Redis sentinel node is deployed on Server B. The server anomaly recovery module is used to quickly resume broadcasting when the server malfunctions. Specifically, it includes: Upon server startup, a Redis monitoring sub-process is initiated, continuously checking the network interface card (NIC) status; NIC status includes connected and disconnected states; when the NIC status changes from disconnected to connected, the server initiates the Redis monitoring sub-process again, continuing to check the NIC status; when the NIC status changes from connected to disconnected, the server sets itself to standby mode, sends a command to stop the Redis monitoring sub-process, and then continues checking the NIC status; the Redis monitoring sub-process checks the local machine status, master node status, and Redis sentinel node information, and performs failover and master-slave node adjustment based on the local machine status when the master node is offline.

2. The high-reliability digital broadcasting system according to claim 1, characterized in that, The Redis monitoring sub-process specifically includes the following steps: S11, Check if the local server status is in standby mode. If yes, stop the local Redis service and proceed to step S12; otherwise, proceed directly to step S12. S12, Check if a command to stop the Redis monitoring sub-process has been received. If yes, stop the Redis monitoring sub-process and enter the stopped state; otherwise, proceed to step S13. S13, Obtain the master node IP, master node status, slave node IP, and Redis sentinel node information. S14, Check if the master node status is offline. If yes, proceed to step S15; otherwise, proceed to step S11. S15, Check if the local server status is working. If yes, proceed to step S16; otherwise, proceed to step S13. S16, Attempt to ping the local Redis node. If the ping fails, start the local Redis service and proceed to step S17; if the ping succeeds, proceed directly to step S17. S17, Notify the local sentinel node to perform failover: sentinel Failover; S18. Check if the failover was successful. If yes, proceed to step S12; otherwise, proceed to step S19; S19. Notify the local Redis node to no longer act as a slave node: slaveof no one; S110. Check if only one sentinel node is deployed on the local machine. If yes, modify the local Redis sentinel service configuration, set the local Redis service as the master node, set the Redis service on the other server as the slave node, and restart the Redis sentinel service. Proceed to step S12; otherwise, proceed directly to step S14; S111. Check if the number of online sentinels is 3 and pinging the local Redis node fails. If yes, start the local Redis service; otherwise, proceed to step S12.

3. The high-reliability digital broadcasting system according to claim 2, characterized in that, The server anomaly recovery module further includes: If the Redis service in server A is the master node and the Redis service in server B is the slave node; if server A disconnects the network cable or shuts down: server A's synchronization service detects that the network card status has become disconnected, the server sets its local machine to standby mode, and server A sends a command to stop the Redis monitoring sub-process, thus stopping the Redis monitoring sub-process; server B detects that the master node is offline, detects that server B's server status is working, and pings its local Redis successfully, then sequentially executes the following: notify the local sentinel node to perform failover: sentinel failover, and notify the local Redis service to no longer act as a slave node: slaveof no First, modify the Redis Sentinel node configuration on server B, setting server B's Redis service as the master node and server A's Redis service as the slave node, and restart server B's Redis Sentinel service to achieve master node failover. When server A reconnects to the network cable or powers on, server A checks if the network card status changes from disconnected to connected. Server A starts the Redis monitoring subprocess. Since server A's server status is in standby mode, the local Redis service is stopped. When server A detects that the master node is online and has been recognized by 3 Sentinel nodes, it restarts server A's Redis service. At this time, the Sentinel will set server A's Redis service as a slave node.

4. The high-reliability digital broadcasting system according to claim 2, characterized in that, The server anomaly recovery module further includes the following steps: If the Redis service in server B is the master node and the Redis service in server A is the slave node, and server B disconnects the network cable or shuts down: Server B checks that the network card status has changed to disconnected, sets its local machine to standby mode, and sends a command to stop the Redis monitoring sub-process. Server A checks that the master node is offline, checks that its local server status is working, and successfully pings its local Redis. It then notifies its local sentinel nodes to perform a failover. Since server A still has two sentinel nodes online, which is more than half, it can successfully elect its local Redis node as the master node. When server B reconnects the network cable or restarts, server B checks that the network card status has changed from disconnected to connected. Server B starts the Redis monitoring sub-process. Since its local server status is standby mode, it stops its local Redis service. When server B monitors that the master node is online and has been recognized by three sentinel nodes, it restarts its local Redis service. At this time, the sentinels will set server B's Redis service as a slave node.

5. The high-reliability digital broadcasting system according to claim 1, characterized in that, The broadcast system maintains a device shadow for each broadcast terminal device through a server, and the broadcast terminal device synchronizes data with the device shadow in the server; the device shadow contains a broadcast module; the broadcast terminal device executes broadcast tasks according to the broadcast control parameters in the broadcast module.

6. The high-reliability digital broadcasting system according to claim 1, characterized in that, The system also includes a database synchronization module for bidirectional data synchronization between the databases on the two servers. The databases on both servers are MySQL databases, and two MySQL master-slave synchronization operations are started in the MySQL database services on the two servers to achieve bidirectional data synchronization.

7. The high-reliability digital broadcasting system according to claim 1, characterized in that, The system also includes a media library synchronization module for bidirectional file synchronization between the media libraries on the two servers; the bidirectional file synchronization includes synchronization of newly added files and synchronization of deleted files.

8. The high-reliability digital broadcasting system according to claim 1, characterized in that, The system also includes a real-time stream synchronization module for broadcasting real-time stream synchronization between two servers.

9. The high-reliability digital broadcasting system according to claim 1, characterized in that, The broadcast terminal device periodically queries the status of the two servers according to a third time interval, and only accepts control messages sent by servers whose status is working; if the server status changes, the server will immediately and proactively send server status information to the terminal device.

10. A method for rapid resuming of high-reliability digital broadcasting in case of an anomaly, based on the high-reliability digital broadcasting system according to any one of claims 1-9, characterized in that, Includes the following steps: S1. Server A loses power, and the server fault recovery module sets Server B as the master node; Server A is the master node, and Server B is the slave node. S2. The broadcast terminal device receives a notification that the working server has changed from Server A to Server B. S3. The broadcast terminal device synchronizes its device shadow from Server B, ensuring the broadcast module data remains consistent with the previous state. S4. The broadcast terminal device requests Server B's relay service to forward the real-time stream, with the request parameters being the stream address and stream ID from the broadcast module. S5. Upon receiving the request, Server B's relay service pulls the RTMP stream from the stream service. S6. Server B's relay service converts the real-time stream into an RTP stream and sends it to the broadcast terminal. S7. Upon receiving the RTP stream, the broadcast terminal device decodes and plays it.

Images