Data stream processing method, device, computer readable storage medium and electronic device

Abstract

The present invention provides a data stream processing method, device, computer-readable storage medium and electronic device, which belong to the field of network technology. In the method, if an update instruction for a target data stream submitted by a first client is received, the computer room subscribing to the target data stream is determined according to the subscription relationship synchronized between each computer room in the distribution system, and the target computer room is obtained. Finally, a data stream update message and updated data of the target data stream are sent to the target computer room, and the data stream update message is used to instruct the target computer room to push updated data to a third client connected to the target computer room. In this way, when a target data stream that needs to be updated appears in any computer room in the distribution system, the updated data and the data stream update message can be directly sent to the target computer room that is subscribing to the target data stream, thereby reducing the delay of data synchronization to a certain extent and improving the efficiency of data synchronization.

Description

Technical Field

[0001] The present invention belongs to the field of network technology, and in particular relates to a data stream processing method, device, computer-readable storage medium and electronic equipment. Background Art

[0002] At present, users in the network are often distributed in different areas. Accordingly, in order to obtain data streams from the network, users in different areas usually connect to different computer rooms through the clients they use to subscribe to data streams for viewing. Correspondingly, users can also connect to different computer rooms through the clients they use to publish data streams.

[0003] In the related art, the computer room that publishes the data stream may be different from the computer room that the subscriber is connected to, and for one data stream, there are many computer rooms that the subscriber is connected to, that is, there are many computer rooms that need to obtain the data stream. Therefore, when updating the data stream to the subscriber, there is often a large delay problem. Summary of the invention

[0004] In view of this, the present invention provides a data stream processing method, device, computer-readable storage medium and electronic device, which solve the problem of large delay to a certain extent.

[0005] According to a first aspect of the present invention, a data stream processing method is provided, which is applied to a first computer room in a distribution system, wherein the first computer room is any computer room in the distribution system, and the method may include:

[0006] If an update instruction for a target data stream submitted by a first client is received, a computer room subscribing to the target data stream is determined according to the subscription relationship synchronized between the computer rooms in the distribution system, and the target computer room is obtained; the subscription relationship is used to indicate the data stream subscribed by the second client connected to each computer room through the computer room;

[0007] A data stream update message and updated data of the target data stream are sent to the target computer room; the data stream update message is used to instruct the target computer room to push the updated data to a third client connected to the target computer room.

[0008] According to a second aspect of the present invention, a data stream processing device is provided, which is applied to a first computer room in a distribution system, wherein the first computer room is any computer room in the distribution system, and the device may include:

[0009] A determination module, configured to determine, upon receiving an update instruction for a target data stream submitted by a first client, a computer room subscribed to the target data stream according to a subscription relationship synchronized between computer rooms in the distribution system, and obtain a target computer room; the subscription relationship is used to indicate a data stream subscribed by a second client connected to each computer room through the computer room;

[0010] The sending module is used to send a data stream update message and the updated data of the target data stream to the target computer room; the data stream update message is used to instruct the target computer room to push the updated data to a third client connected to the target computer room.

[0011] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the data stream processing method described in the first aspect are implemented.

[0012] In a fourth aspect, the present invention provides an electronic device comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the data stream processing method described in the first aspect when executing the program.

[0013] Compared with the prior art, the present invention has the following advantages:

[0014] The data stream processing method provided by the embodiment of the present invention, if receiving the update instruction of the target data stream submitted by the first client, according to the subscription relationship synchronized between each computer room in the distribution system, the computer room subscribing to the target data stream is determined, and the target computer room is obtained, and the subscription relationship is used to indicate the data stream subscribed by the second client connected to each computer room through each computer room. Finally, a data stream update message and the update data of the target data stream are sent to the target computer room, and the data stream update message is used to indicate that the target computer room pushes the update data to the third client connected to the target computer room. In this way, through the synchronized subscription relationship between the computer rooms in the distribution system, any computer room in the distribution system can know in advance the data stream subscribed by the client connected to other computer rooms through the other computer rooms. Accordingly, when any computer room in the distribution system has a target data stream that needs to be updated, the update data and the data stream update message can be directly sent to the target computer room that is subscribing to the target data stream, so that the target computer room can quickly push updates to the third client subscribing to the target data stream through the target computer room, thereby reducing the delay of data synchronization to a certain extent and improving the efficiency of data synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0016] Figure 1 is a flow chart of steps of a data stream processing method provided by an embodiment of the present invention;

[0017] Figure 2 is a schematic diagram of a distribution system provided by an embodiment of the present invention;

[0018] Figure 3 It is an example diagram of a local connection and file structure provided by an embodiment of the present invention;

[0019] Figure 4 is another example diagram of a local connection and a file structure provided by an embodiment of the present invention;

[0020] Figure 5 It is another example diagram of a local connection and a file structure provided by an embodiment of the present invention;

[0021] Figure 6 is a block diagram of a data stream processing device provided by an embodiment of the present invention;

[0022] Figure 7 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0024] First, the basic functions supported by the distribution system in the embodiment of the present invention are described. In the distribution system, the client can access the distribution system through a preset connection command (e.g., Connect (SessionId) command). Among them, the same client carries the same client identifier (SessionId) when accessing at different times. The client can also specify a path (Path) through a preset command (e.g., Set (Path, Data)) command, and create / modify the content of the corresponding Session file under the directory of the Path, that is, publish / update the data stream. Alternatively, the Session file under the specified Path can also be deleted through a preset delete command (e.g., Delete (Path) command), and the data changes of the directory under the Path can be specified through a preset subscription command (e.g., Subscribe (Path, Depth) command), and the depth of the monitoring can be defined to achieve subscription. The monitoring is canceled through a preset unsubscribe command (e.g., UnsubScribe (Path, Depth) command) to unsubscribe. Further, the distribution system can also push data change events to the client through a preset notification command (e.g., Notify (Path, Version, Data) command).

[0025] Furthermore, in an implementation scenario, for example, in a data stream subscription scenario, currently, subscription is often performed across computer rooms, that is, clients are distributed in different computer rooms and subscribe through different computer rooms. Since the distance between computer rooms is often relatively far, for example, the round-trip time (RTT) between computer rooms is sometimes greater than 100 milliseconds (ms). Therefore, how to enable clients to quickly subscribe to data changes submitted by other clients through different computer rooms, that is, how to quickly synchronize the updated data of the data stream submitted by the publishing method to the subscription method, has become an issue of increasing concern. To this end, an embodiment of the present invention proposes a data stream processing method applied to a distribution system.

[0026] The data stream processing method in the embodiment of the present invention is described in detail below.

[0027] Embodiment 1

[0028] Figure 1 is a flow chart of steps of a data stream processing method provided by an embodiment of the present invention. The method can be applied to a first computer room in a distribution system. The first computer room is any computer room in the distribution system, such as Figure 1 As shown, the method may include:

[0029] Step 101: If an update instruction for a target data stream submitted by a first client is received, the computer room subscribed to the target data stream is determined according to the subscription relationship synchronized between each computer room in the distribution system, and the target computer room is obtained; the subscription relationship is used to indicate the data stream subscribed by the second client connected to each of the computer rooms through the computer room.

[0030] In an embodiment of the present invention, a distribution system can be used to implement the publishing and subscription of data streams. The distribution system can be composed of multiple computer rooms, which can support clients to publish or subscribe to data streams. A computer room can be composed of one or more servers. Among them, publishing can also be called publish-subscribe. In this distribution system, publishing can mean that the client creates a temporary directory with data under a specified path in the computer room in the distribution system, for example, a data stream directory. Each published data stream can create a corresponding data stream directory. Subscription can mean that the client specifies through the computer room to monitor a temporary directory under a certain path, and specifically can monitor the data changes in the temporary directory, for example, it can monitor the file changes under the temporary directory.

[0031] Further, the first client may be a client connected to the first computer room and publishing the target data stream through the first computer room, the target data stream may be any data stream published through the first computer room, and the update instruction may be used to change the content of the target data stream, for example, the update instruction may be used to add data to the target data stream. The second client may be any client connected to any computer room in the distribution system, and the subscription relationship may be used to indicate the data stream subscribed by the second client connected to each computer room through the computer room. For example, the subscription relationship may be a corresponding relationship between each computer room and the data stream subscribed by the computer room. Among them, the data stream subscribed by the computer room may be subscribed in response to the subscription request of the second client, and the data stream subscribed by the second client through the computer room may be published by the computer room or by other computer rooms. Accordingly, when the first client in the first computer room needs to make data changes to the target data stream it publishes, the first computer room may directly search for the computer room in the distribution system that has subscribed to the target data stream based on the subscription relationship as the target computer room.

[0032] Step 102: Send a data stream update message and updated data of the target data stream to the target computer room; the data stream update message is used to instruct the target computer room to push the updated data to a third client connected to the target computer room.

[0033] In an embodiment of the present invention, the update data may be the required newly added data indicated by the update instruction, and the update data may be sent to the first computer room together with the update instruction. Furthermore, the first computer room of the distribution system may establish a connection with each other computer room in advance, so that the sending time may be reduced to a certain extent. When sending, the first computer room may directly push the data stream update message and the update data to the target computer room, so that the target computer room may directly push to the third client based on the received content to ensure the push efficiency. Among them, the third client may be a client connected to the target computer room, which subscribes to the target data stream through the target computer room. Alternatively, in an embodiment of the present invention, the data stream update message may be sent first, and after receiving the data stream update message, the target computer room may actively request the first computer room to pull the update data, and then the first computer room may send the update data, and the present invention does not limit this.

[0034] In summary, the data stream processing method provided by the embodiment of the present invention, if receiving the update instruction of the target data stream submitted by the first client, according to the subscription relationship synchronized between each computer room in the distribution system, the computer room subscribing to the target data stream is determined, and the target computer room is obtained, and the subscription relationship is used to indicate the data stream subscribed by the second client connected to each computer room through each computer room. Finally, a data stream update message and the updated data of the target data stream are sent to the target computer room, and the data stream update message is used to indicate that the target computer room pushes the updated data to the third client connected to the target computer room. In this way, through the synchronized subscription relationship between the computer rooms in the distribution system, any computer room in the distribution system can know in advance the data stream subscribed by the client connected to other computer rooms through the other computer rooms. Accordingly, when any computer room in the distribution system has a target data stream that needs to be updated, the update data and the data stream update message can be directly sent to the target computer room that is subscribing to the target data stream, so that the target computer room can quickly push updates to the third client subscribing to the target data stream through the target computer room, thereby reducing the delay of data synchronization to a certain extent and improving the efficiency of data synchronization.

[0035] Embodiment 2

[0036] Optionally, an instance may be deployed in the first computer room, and each instance deployed in the first computer room is connected to at least one instance in another computer room. The instance in the embodiment of the present invention may be a SubscribeCenter Server (SCS) instance. Accordingly, the operation of sending a data stream update message and the updated data of the target data stream to the target computer room may specifically include:

[0037] Step S21: Send the data stream update message and the update data to the connected instances in the target computer room through each of the instances in the first computer room.

[0038] In the embodiment of the present invention, the instance can be run as an independent process in the computer room, and the connection between the instances can be established when the distribution system is initialized. Since each instance deployed in the first computer room has established a connection with at least one instance in the other computer room, when sending through each instance in the first computer room, the data stream update message and the update data can be directly sent to the instance in the target computer room connected to the instance in the first computer room based on the pre-established connection. In this way, since there is no need to perform the operation of establishing a connection, the data stream update message and the update data can be sent directly based on the pre-connected instances between the computer rooms, thereby ensuring the data transmission efficiency to a certain extent, thereby reducing the delay.

[0039] Embodiment 3

[0040] Optionally, in the embodiment of the present invention, at least two instances are deployed in the first computer room, that is, each computer room of the distribution system can be deployed in multiple points, and the number of instances deployed in each computer room of the distribution system is not less than 2. Accordingly, after receiving the above update instruction, the embodiment of the present invention can also perform the following steps:

[0041] Step S31: Transmit the updated data to each of the instances in the first computer room, so as to store the updated data locally in each of the instances.

[0042] In an embodiment of the present invention, a software development kit (SDK) service may also be deployed in the first computer room, and each instance deployed in the first computer room may be connected to the SDK service. Accordingly, in this step, update data may be input to all instances in this computer room through the SDK service. Specifically, a data update command carrying update data may be sent to all instances, for example, a Set command may be sent to control each instance to save the update data locally. Since multiple instances are deployed in the first computer room, multiple copies of update data may be stored in the first computer room through this step, thereby providing a basis for disaster recovery. Furthermore, at least two instances in the first computer room may be distributed in at least two servers in the computer room. In this way, the problem that data cannot be provided when a server fails due to at least two instances being in the same server may be avoided, thereby further improving disaster recovery capabilities.

[0043] Step S32: in response to a connection request sent by the first instance in the second computer room, reestablish a connection between the first instance and a non-faulty instance in the first computer room; the connection request is sent by the first instance when an instance connected to the first computer room fails.

[0044] In this step, the connection request may be sent by the first instance when all instances connected in the first computer room fail, the second computer room may be a computer room including the first instance in other computer rooms, and the first instance may be an instance when all instances connected in the first computer room fail. For example, assuming that instance 1 in the first computer room fails, and instance 4 in computer room 3 is only connected to instance 1 in the first computer room, then computer room 3 is the second computer room, and instance 4 is the first instance in the second computer room. In the embodiment of the present invention, when the instance connected to the first instance in the second computer room in the first computer room fails, that is, when the first instance in the second computer room is not connected to an instance in the first computer room, re-establishing the connection between the first instance and the non-faulty instance in the first computer room can ensure that each instance in each computer room is connected to at least one instance in other computer rooms, thereby ensuring that data can be conveniently sent and received between computer rooms.

[0045] Step S33: Synchronize the updated data locally stored in the non-faulty instance to the first instance based on the connection.

[0046] In this step, the updated data locally stored in the non-faulty instance on the new connection of the first instance can be synchronized to the first instance through the re-established connection, thereby avoiding the problem of low system availability due to node abnormality.

[0047] It should be noted that if a deletion instruction for the target data stream submitted by the first client (for example, a delete instruction) is received, a data stream deletion message can also be sent to the target computer room in an embodiment of the present invention. The data stream deletion message can be used to instruct the target computer room to delete the relevant data of the target data stream, thereby avoiding the storage of unnecessary data in the target computer room, which in turn leads to the problem of wasted storage space.

[0048] In the embodiment of the present invention, it is only necessary to transmit the update data to each instance in the first computer room so as to save the update data locally in each instance, so as to provide a disaster recovery basis for the distribution system and improve the availability of the distribution system. Therefore, the implementation cost of high availability of the distribution system can be reduced to a certain extent. Further, in response to the connection request sent by the first instance in the second computer room, the connection between the first instance and the non-faulty instance in the first computer room is re-established; the connection request is sent by the first instance when the instance connected in the first computer room fails. The update data saved locally in the non-faulty instance is synchronized to the first instance based on the connection. In this way, by deploying multiple instances in each computer room, and saving the update data in each instance when data update occurs, that is, storing multiple copies of the update data. When the instance in the first computer room fails later, other non-faulty instances can be connected to the first instance in the second computer room, so that the data update provided to the second computer room continues through the non-faulty instance, so that the reliability of the distribution system can be ensured while ensuring the efficiency of synchronous update to a certain extent, thereby improving the availability of the distribution system.

[0049] Embodiment 4

[0050] Optionally, in the embodiment of the present invention, the operation of transmitting the updated data to each of the instances in the first computer room so as to locally store the updated data in each of the instances may specifically include:

[0051] Step S41: Setting version information for the update data.

[0052] In this step, the version information (version) may be set by the SDK service in a sequentially increasing manner, wherein the version information set for each updated data submitted by a client may be different. The version information may be a version number, and the SDK service may generate a locally increasing version number. That is, a globally increasing version number may be maintained within the SDK service, and each time the data is updated, regardless of which file, that is, regardless of which data stream, the version number will be increased by one. Accordingly, the client can only create, delete, or modify files it has created, and cannot modify files created by other clients. Therefore, by maintaining a globally increasing version number, the consistency of the file content (that is, updated data) published by the client can be ensured by the version number.

[0053] It should be noted that the distribution system of the embodiment of the present invention can organize data in a structure similar to a file system, and when publishing data, a temporary file with data can be created in a path specified by a preset data model, thereby publishing data. The data model can be expressed as:

[0054] / path / by / client / defined:

[0055] -{sessionid}:{data} <version>

[0056] -{sessionid}:{data} <version>

[0057] The path can be customized by the business, a sessionid uniquely represents a client, data can represent specific business data (ie, data in the data stream), version can represent the version number, and the version number can be controlled to increase each time the data is updated (ie, a version++ operation is performed).

[0058] Step S42: Transmit the update data and the version information to each of the instances in the first computer room, and store the update data locally in the instance through the instance if the version information is higher than the version information currently held by the instance.

[0059] In the embodiment of the present invention, if the version information is higher, it can be said that the updated data is newer. Therefore, when updating and storing according to the version information, it can be ensured that the data with a higher version will not be overwritten by the data with a lower version, thereby ensuring the accuracy of data storage. Specifically, a data update command carrying the updated data, the current version information and the identifier of the first client (for example, sessionid) can be sent to the instance, and the instance can perform a local storage operation in response to the data update command. For example, it can be stored in the cache corresponding to the instance. That is, when the updated data submitted by the first client is synchronized with the instance in the first computer room, the monotonically increasing version number and sessionid will be carried at the same time. The instance in the first computer room will determine whether the data it holds is consistent with the data submitted by the first client this time based on the received version number and sessionid, that is, the data of the target data stream currently released by the first client is determined based on the sessionid, and then the received version number and the version number of the held data are determined to be the same. If they are the same, it can be considered that the data update command received this time is triggered by mistake, and local storage can be omitted. On the contrary, if they are not the same and the version number is higher than the version number of the currently held data, the local storage operation can be performed.

[0060] It should be noted that the instance can further save the version number and sessionid in correspondence, and when sending data stream update messages and update data to the instance connected in the target computer room, it can be further sent to the instance connected in the target computer room. Correspondingly, the instance in the target computer room that receives this information can also locally store the update data based on the version number and sessionid to avoid the problem of low-version data overwriting high-version data due to false triggering. That is, when synchronizing data between instances, when synchronizing instances between different computer rooms, the data consistency between the two SCS nodes will also be guaranteed based on the version number. Among them, the method of locally storing updated data based on the version number and sessionid refers to the aforementioned related description and will not be repeated here.

[0061] Furthermore, when pushing to the SDK service of the target computer room that subscribes to the target data stream, the version number and sessionid can also be carried at the same time. In this way, when the SDK service of the target computer room pushes updated data to the third client, it can first update the local data according to the version number and sessionid to ensure that the data with a larger version number will not be overwritten by the data with a smaller version number, and then push the local data to the third client, thereby ensuring that the updated data pushed to the third client is the latest data.

[0062] In the embodiment of the present invention, version information is set for the updated data, the updated data and version information are transmitted to each instance in the first computer room, and when the version information is higher than the version information currently held by the instance, the updated data is stored locally in the instance. In this way, the mechanism of setting a version number for the data content can ensure the final consistency of the data stored locally in each instance through the version number, avoiding the problem of incorrectly sending data update commands due to false triggering, thereby causing abnormal data updates.

[0063] Embodiment 5

[0064] Optionally, the subscription relationship in the embodiment of the present invention may be synchronized through the following steps:

[0065] Step S51: upon receiving a subscription request sent by a fourth client, generating a subscription relationship corresponding to the subscription request; the subscription relationship is used to represent the corresponding relationship between the first computer room and the data stream subscribed by the fourth client through the subscription request.

[0066] In this step, the fourth client can be any client connected to the first computer room, and the subscription request (for example, a subscribe request) can be used to subscribe to the data stream published by the computer room in the distribution system through the first computer room. Accordingly, when the subscription request is received, the subscription request can be parsed to determine the data stream to which the fourth client subscribes through the subscription request. For example, the subscription request can carry the live broadcast room number, and accordingly, the data stream in the live broadcast room corresponding to the live broadcast room number can be determined as the data stream subscribed through the subscription request. Accordingly, the first computer room can locally record the correspondence between the identifier of the fourth client and the identifier of the data stream, so as to facilitate the subsequent push of the data stream to the fourth client. Furthermore, the correspondence between the identifier of the first computer room and the identifier of the data stream can be generated to obtain the subscription relationship corresponding to the subscription request.

[0067] Step S52: Synchronize the subscription relationship to instances in any other computer room connected to the instance through each of the instances in the first computer room.

[0068] In this step, since the instance in the first computer room is connected to the instance in any other computer room, the subscription relationship can be easily synchronized to the other computer rooms. Furthermore, since the first computer room can be any computer room in the distribution system, through the above steps S51 to S52, each computer room in the distribution system can obtain the current subscription relationship of other computer rooms, thereby ensuring the efficiency of subsequent update data forwarding.

[0069] It should be noted that in an application scenario, it is often necessary to synchronize data between servers in different computer rooms in a distributed system, such as configuration, status, etc. For example, in a streaming media service, users in a live broadcast room are often distributed in different countries and regions, and users connect to different computer room servers for streaming or pulling. When a user connects to a server node and hopes to pull the streaming data of other users in a room, the server to which the user is connected often needs to know the server to which other current streaming users are connected, and establish a connection to the corresponding server and pull the stream. How to quickly and accurately synchronize user and room information between different servers directly affects the user experience. Therefore, in an embodiment of the present invention, a publishing relationship corresponding to a publishing request can also be generated when a publishing request is received; the publishing relationship is used to characterize the corresponding relationship between the data stream published by the first computer room and the fourth client through the publishing request. Further, the publishing relationship can be synchronized to the instance in any other computer room connected to the instance through each instance in the first computer room, so that each computer room of the distribution system can obtain the current publishing relationship of other computer rooms, thereby ensuring that when the data of a certain data stream needs to be actively obtained in the future, the pulling of the stream can be conveniently realized based on the publishing relationship, thereby ensuring the acquisition efficiency.

[0070] Furthermore, in an embodiment of the present invention, when the instance in the first computer room stores relevant data of the data stream to which the subscription request is subscribed, that is, the first computer room currently stores data of the data stream locally, the relevant data of the data stream can be directly pushed to the third client to ensure subscription efficiency. Among them, the data of the data stream currently stored locally in the first computer room can be stored when returning data to other clients that subscribe to the data stream through the first computer room, or can be stored locally in the first computer room when other clients publish the data stream through the first computer room, and the present invention does not limit this. Of course, when the first computer room does not store relevant data of the data stream locally, the computer room that publishes the data stream is determined based on the currently received publishing relationship, and the relevant data of the data stream is obtained from the computer room. Finally, the relevant data can be pushed to the fourth client based on the SDK service in the first computer room.

[0071] In an embodiment of the present invention, when a subscription request sent by a fourth client is received, a subscription relationship corresponding to the subscription request is generated, and the subscription relationship is used to characterize the corresponding relationship between the first computer room and the data stream subscribed by the fourth client through the subscription request. Through each instance in the first computer room, the subscription relationship is synchronized to an instance in any other computer room connected to the instance. In this way, by synchronizing the subscription relationship in the distribution system in real time when a subscription request is received, other computer rooms can grasp the subscription relationship in a timely manner, and when the data stream subscribed by the subscription request is updated, the updated data can be synchronized to the computer room in a timely manner, thereby ensuring synchronization efficiency.

[0072] Embodiment 6

[0073] Optionally, the following steps may also be performed in the embodiment of the present invention:

[0074] Step S61: upon receiving an unsubscribe request sent by the fifth client, deleting the subscription relationship corresponding to the unsubscribe request, and synchronizing the unsubscribe request to instances in any other computer room connected to the instance through the instances in the first computer room.

[0075] Among them, the fifth client may be a client that subscribes to the data stream through the first computer room, and the unsubscribe request (for example, Unsubscribe request) may be sent by the fifth client when the subscription needs to be unsubscribed. For example, when a user needs to exit the live broadcast room and unsubscribe from the data stream of the live broadcast room, it may be sent to the first computer room through the fifth client held by the user. The subscription relationship corresponding to the unsubscribe request may be the subscription relationship generated when requesting to subscribe to the data stream indicated by the unsubscribe request. Accordingly, in the case of receiving the unsubscribe request, the subscription relationship corresponding to the unsubscribe request stored locally can be deleted accordingly. At the same time, by further synchronizing the unsubscribe request to an instance in any other computer room connected to each instance in the first computer room, the other computer rooms can respond to the unsubscribe request and delete the subscription relationship stored locally accordingly. Of course, further, each computer room can delete the relevant data of the data stream indicated by the unsubscribe request stored this time to release storage space and avoid space waste.

[0076] In the embodiment of the present invention, when receiving the unsubscribe request sent by the fifth client, the subscription relationship corresponding to the unsubscribe request is deleted, and the unsubscribe request is synchronized to the instances in any other computer room connected to the instance through each instance in the first computer room. In this way, it is avoided that the computer room in the distribution system stores unnecessary subscription relationships, which increases the difficulty of subsequent search and determination of the target computer room and wastes local storage space.

[0077] Embodiment 7

[0078] Optionally, the distribution system in the embodiment of the present invention may further include a global module, which may store relevant information of the instances deployed in each computer room. For example, the global module may be a globally unique Daemon module, which may be used to manage the ring (Topo) structure formed by the instances inside the subscription center constituted by the distribution system. Among them, the Topo structure may be composed of instances interconnected between various computer rooms. During the service deployment phase, the instances in the first computer room may register with the global module after startup, that is, the instances in each computer room in the distribution system may perform a registration operation during the startup phase, so that the global module may obtain relevant information of each computer room and the instances deployed therein, that is, the global module may grasp the instance deployment status in each computer room. Furthermore, the embodiment of the present invention may also perform the following operations:

[0079] Step S71: Obtain relevant information of the instances deployed in each of the other computer rooms from the global module.

[0080] In the embodiment of the present invention, a request may be sent to the global module to obtain information about each of the other computer rooms and the instances deployed therein, so as to learn the instance deployment status in the other computer rooms. The instance-related information may be information required to establish a connection, for example, the running address and identifier of the instance.

[0081] Step S72: For any of the instances in the first computer room, a long connection is established between the instance and an instance in each of the other computer rooms according to relevant information of the instances deployed in each of the other computer rooms.

[0082] In this step, for any instance in the first computer room, the instance can select an instance from the instances in other computer rooms, and establish a long connection with the instance based on the relevant information of the selected instance, that is, establish a TCP connection keep-alive, so as to ensure that each instance in the distribution system is connected to an instance in other computer rooms. By establishing a long connection, the connection quality can be ensured, so as to ensure that data can be sent and received stably based on the connection between the two in the future.

[0083] In the embodiment of the present invention, the relevant information of the instances deployed in each other computer room is obtained from the global module. For any instance in the first computer room, a long connection between the instance and an instance in each other computer room is established according to the relevant information of the instances deployed in each other computer room. In this way, the required connection resources can be saved to a large extent while realizing the interconnection between the computer rooms in the distribution system.

[0084] Embodiment 8

[0085] Optionally, the following operations may also be performed in the embodiment of the present invention:

[0086] Step S81: If a directory creation instruction is received from a second instance in another computer room, a receiving directory corresponding to the data stream directory indicated by the directory creation instruction is created through a third instance in the first computer room, and M tag files corresponding to the receiving directory are created; the receiving directory is used to store relevant data of the data stream published under the data stream directory, and the M is the number of the second instances, and the third instance is connected to the second instance.

[0087] In this step, the directory creation instruction can be used to instruct the first computer room to locally create a receiving directory corresponding to the data stream directory indicated by the directory creation instruction. Among them, a data stream directory can correspond to a data stream, and the directory creation instruction can be sent by the first computer room after requesting to subscribe to the data stream corresponding to the data stream directory indicated by the directory creation instruction, and the data stream directory is created in the computer room where the second instance is located. By locally creating a receiving directory corresponding to the data stream directory indicated by the directory creation instruction, the first computer room can store the relevant data in the receiving directory after obtaining the relevant data of the data stream published under the data stream directory.

[0088] Accordingly, based on the internally deployed SDK service, the data in the receiving directory can be pushed to the client subscribing to the data stream through the first computer room. The receiving directory can be created under the root directory. The data stored in the receiving directory can be stored in the form of files.

[0089] Further, in the embodiment of the present invention, when a directory creation instruction is received, the operation of creating a receiving directory can be executed, so that the operation of creating a receiving directory can be realized with only one second instance confirmation. Since only the synchronization message of one SCS instance needs to be received, when a new directory or file is created, it can be synchronized to other terminals as soon as possible, thereby improving the subscription efficiency to a certain extent. After creating the receiving directory, M markup files corresponding to the receiving directory can be further created. The markup file can be a soft connection file, and the soft connection file can point to the receiving directory. That is, when the instance in the first computer room is connected to multiple second instances in other computer rooms, the multiple second instances in the other computer rooms push the directory creation instruction to the instance connected in the first computer room, and accordingly, multiple soft connection files can be created in the end. It should be noted that when the client creates a directory through the SCS instance, the SCS instance can also create a corresponding soft connection file for easy maintenance. Further, the receiving directory in the embodiment of the present invention can be created as other application directories under the root directory. A special subdirectory (for example, .scs directory) can be pre-set under the root directory. This subdirectory can be used to save the information of the scs instance. Each subdirectory under .scs can be used to represent the peer SCS instance directly connected to the SCS instance. In a specific scs directory, soft link files can be saved, and the soft link files can correspond to the file paths one by one. For example, / .scs / scs04 / appid01 / edge / sid02 / idc04 points to / appid01 / edge / sid02 / idc04. It should be noted that when the long connection between the SCS instance and some other SCS instance is disconnected, all soft link files created by the SCS instance can be cleared accordingly to avoid wasting space. Furthermore, if the number of files or subdirectories under a directory is 0, the directory can be automatically deleted to optimize the directory structure.

[0090] Step S82: upon receiving a deletion instruction sent by any of the second instances, deleting one of the marked files through the third instance, and deleting the receiving directory when all the marked files are deleted.

[0091] In this step, the deletion instruction may be sent after the second instance deletes the data under the data stream directory indicated by the directory creation instruction stored locally. Accordingly, after receiving the deletion instruction, compared with the method of directly deleting the receiving directory, the embodiment of the present invention will first delete the marking file, for example, the soft link file, and will finally delete the receiving directory when all marking files are deleted, that is, when all second instances are confirmed. In other words, during the subscription process, the operation of deleting the receiving directory requires confirmation from multiple SCS instances, thereby avoiding the problem of accidental deletion to a certain extent and ensuring the security of data.

[0092] Embodiment 9

[0093] The following is an explanation in conjunction with an application scenario of the present invention. In a related technology, the purpose of data synchronization is often achieved through a middleware that supports publishing and subscription. Each server centrally updates the user room information connected to it to the middleware, and then the subscription center synchronizes the information about the room synchronized by other servers to the server. Specifically, by building a middleware, all clients / servers submit data to this middleware, and subscribe / publish data through this middleware. For example, in one way, it is often implemented through open source components, such as by building a Redis cluster, all clients connect to the Redis service through the Redis protocol, so as to achieve the data change of the corresponding value under the specified key, or build an etcd cluster, and through the Watch mechanism of Etcd, the purpose of synchronizing subscription data can also be achieved. However, in the way of building etcd, since data needs to be synchronized between multiple nodes in Etcd, each data update will be submitted only after synchronization confirmation between more than half of the Etcd nodes. Therefore, when deployed across computer rooms, there will be a high update delay, which cannot meet the business with high requirements for timeliness. Furthermore, when a node is abnormal or the network is unavailable, the solution of the related art may cause a short period of service unavailability. For example, when a Redis master node is abnormal or the network link is abnormally disconnected, the recovery mechanism of Redis takes a long time, for example, at least the duration of a heartbeat detection is required. Therefore, the availability of the service cannot be guaranteed.

[0094] Take the distribution system including 4 computer rooms as an example. Figure 2 is a schematic diagram of a distribution system provided by an embodiment of the present invention, such as Figure 2 As shown, each computer room deploys multiple SCS instances and SDK services. Among them, the SCS instance can be used to receive, cache and forward messages / data. After initialization, the client (i.e., the "Client" in the figure) can connect to all SCS instances in the computer room through the SDK server to perform operations such as sending subscription commands to the SCS instance and receiving messages from the SCS instance. Specifically, the client can subscribe to data changes. When the data in the temporary directory under the subscribed path changes, for example, after creating or deleting a file, the client that has subscribed to the path of the temporary directory can be notified in time to inform the file changes in the directory. When the client actively deletes or disconnects, the files it created can be randomly deleted to save storage space.

[0095] Further, Figure 2 The "Daemon" in the figure indicates a global module. Daemon can be used for service discovery and scheduling between SCS instances and between Client and SCS instances to ensure that Client is connected to SCS instances and SCS instances in different computer rooms. In actual scenarios, SDK services can connect to all SCS instances in the computer room through Daemon (the figure only shows some of the connections between Daemon in the computer room). SCS instances can select an SCS instance in a computer room other than the computer room and establish a TCP connection keep-alive to ensure that each SCS instance has at least one connection with other computer rooms.

[0096] During the data subscription stage, subscription-related commands, such as Subscribe / Unsubscribe commands, can be sent to all SCSs in the computer room through the SDK service. Accordingly, in the subscription scenario, after the SCS instance receives the command, if there is corresponding data locally, it can be pushed directly to the SDK service, so as to be pushed to the corresponding client through the SDK service. At the same time, subscription messages can be synchronized to all connected SCSs to synchronize subscription relationships. Among them, when synchronizing subscription relationships, it can be synchronized to all connected SCSs, whether they are actively connected or passively connected, and the subscription relationship can be synchronized based on a broadcast mechanism.

[0097] In this way, during the data update phase, the SCS instance can save the updated data locally and push the data update message to the connection that has subscribed to the command, that is, execute the operation of sending the data stream update message and the updated data of the target data stream to the target computer room. Correspondingly, the SCS instance in the target computer room can synchronize to the corresponding Client (that is, push the updated data to the third client) after receiving the updated data. Since the SCS instance in the first computer room will forward the updated data to all subscribed computer rooms, the SCS instance in the target computer room can only perform the operation of synchronizing to the corresponding Client without having to continue forwarding to other SCSs, thereby saving processing resources to a certain extent. It should be noted that the data transmission involved in the distribution system in the embodiment of the present invention can all be based on memory transmission, and the data will not be persisted and saved, thereby avoiding the problem of excessive space occupation. Further, the embodiment of the present invention can be implemented through background service design, so that while improving the performance and reliability of data synchronization, it can avoid changes and influences on the functions and interaction methods of products implemented based on the distribution system, thereby reducing the application difficulty of the distribution system to a certain extent.

[0098] For streaming media services that have strong real-time performance and can sacrifice certain consistency in extreme cases, that is, weak consistency can be tolerated, and the system is allowed to have data inconsistencies for a short period of time. In the embodiment of the present invention, combined with the needs of streaming media services, the subscription relationship will be synchronized in advance between computer rooms. When there is data update, the data update can be quickly synchronized to the subscribed computer room node through the subscription relationship. Specifically, when there is relevant data update, it will be immediately forwarded by the SCS instance in the first computer room to other subscribed computer rooms. The delay of data forwarding only depends on a round-trip time between the two computer rooms, and there will be no other time consumption. Compared with other implementation methods, the flow of data within the distribution system basically does not generate additional time consumption. In this way, the data synchronization speed can be the fastest by increasing subscription broadcasts, thereby meeting the needs of fast synchronization of subscription data. At the same time, by synchronizing the updated data to multiple SCS instances in the computer room at the same time, other SCS instances can be used to continue to provide data when an exception occurs, thereby improving availability. In this way, a highly available and high-performance subscription center service can be constructed, thereby taking into account the availability of the service, the timeliness of data synchronization, and the effectiveness.

[0099] Embodiment 10

[0100] In an application scenario, the SCS instance and the SDK service may both maintain a data structure similar to the directory structure of a file system. Figure 3 is a diagram of a local connection and file structure example provided by an embodiment of the present invention, such as Figure 3 As shown in the figure, Client2 is subscribing to the changes of the first-level subdirectories under the / app01 / sid02 directory through the SCS3 instance and the SCS4 instance. At this time, the directory structure saved locally on each node can be shown in the figure. Specifically, Client1 creates its own temporary files under / app01 / sid01 / cn and / app01 / sid02 / cn, and Client2 creates its own temporary files under / app01 / sid03 / as. Since Client2 subscribes to the changes under / app01 / sid02, it will synchronize the information of the / app01 / sid02 / cn subdirectory.

[0101] Furthermore, when the SCS4 instance crashes or stops, the local directory structure of Client2 will change to:

[0102] - / app01 / sid03 / as / client2

[0103] - / app01 / sid02 / cn

[0104] - / .scs / scs3 / app01 / sid02 / cn

[0105]

[0106] However, because SCS3 maintains the data and does not delete it, abnormal situations in the SCS4 instance will not affect the structural changes of the application directory under the root directory, thereby ensuring that the application directory structure (the first two lines of the client directory) remains consistent for the client, ensuring that when an SCS instance has an abnormality, it will not affect the clients connected to each computer room, and ensuring that the subscribed status on the client remains consistent.

[0107] Furthermore, if the SCS4 instance has not been restarted, the SCS1 instance will establish a connection with the SCS3 instance. After the SCS1 instance and the SCS3 instance are connected, the SCS3 instance can subscribe to the relevant directory from the SCS1 instance. Accordingly, without affecting the change of the application directory structure, the local directory structure of the SCS3 instance can add a reference file (i.e., soft link file) to change to the following structure:

[0108] - / app01 / sid03 / as / client2

[0109] - / app01 / sid02 / cn

[0110] - / .scs / scs2 / app01 / sid02 / cn

[0111] - / .scs / scs1 / app01 / sid02 / cn

[0112] Since SCS3 maintains the data and does not delete it, abnormal situations in the SCS4 instance will not affect the structural changes of the application directory under the root directory, and the SCS1 instance will re-establish the connection with the SCS3 instance, so that computer room 1 can continue to provide updated data to computer room 2, thereby ensuring the availability of the distribution system.

[0113] Of course, after the SCS4 instance is restarted, it can reconnect to the SCS1 instance, and then Client2 can synchronize local data and directories that need to be subscribed to the SCS4 instance. The SCS1 instance can then synchronize to the SCS4 instance again, thereby realizing the recovery of the SCS4 instance without affecting the application directory structure of Client2.

[0114] Further, Figure 4 This is another example diagram of local connection and file structure provided by an embodiment of the present invention. Assuming that the SCS1 instance crashes, the local structure of each node may be as follows: Figure 4 However, since the changes will not affect the application directory structure on Client2, after the SCS1 instance is restarted, the data can be restored to the state before the SCS1 instance crashed through synchronization between Client1 and SCS4 instances. Figure 5 This is another example diagram of a local connection and file structure provided by an embodiment of the present invention. Assuming that the connection between Client1 and SCS2 instance is disconnected due to a network anomaly, the local structure of each node may be as follows: Figure 5 However, based on the same reasons mentioned above, it can also be ensured that the application directory structure on Client2 (the first two rows of directories on Client2) will not be affected, ensuring the availability of the distribution system.

[0115] In summary, in the embodiment of the present invention, by deploying SCS instances at multiple points in one computer room, when the client sends data to the distribution system, multiple copies are simultaneously sent to the SCS instances in the first computer room, and the data is forwarded simultaneously through multiple links, ensuring that any single point abnormality will not affect the forwarding efficiency. That is, if one of the SCS instances has an abnormality such as downtime or network disconnection, data synchronization can be performed through another SCS instance to ensure that the forwarding of the subscribed data will not be affected, thereby eliminating the unavailable time of seconds caused by system abnormalities to a certain extent, thereby improving service availability.

[0116] Embodiment 11

[0117] Figure 6 is a block diagram of a data stream processing device provided by an embodiment of the present invention, which is applied to a first computer room in a distribution system, wherein the first computer room is any computer room in the distribution system, such as Figure 6 As shown, the device 20 may include:

[0118] The determination module 201 is used to determine the computer room subscribed to the target data stream according to the subscription relationship synchronized between the computer rooms in the distribution system, and obtain the target computer room if an update instruction for the target data stream submitted by the first client is received; the subscription relationship is used to indicate the data stream subscribed by the second client connected to each computer room through the computer room;

[0119] The sending module 202 is used to send a data stream update message and the updated data of the target data stream to the target computer room; the data stream update message is used to instruct the target computer room to push the updated data to a third client connected to the target computer room.

[0120] The data stream processing device provided in the embodiment of the present invention has a functional module corresponding to executing the data stream processing method, and can execute the data stream processing method provided in any one of Embodiments 1 to 10 of the present invention, and can achieve the same beneficial effects.

[0121] In another embodiment of the present invention, an electronic device is provided. The electronic device may include: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the program, each process of the above data stream processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not described here. For example, Figure 7 As shown, the electronic device may specifically include: a processor 401, a storage device 402, a display screen 403 with a touch function, an input device 404, an output device 405, and a communication device 406. The number of processors 401 in the electronic device may be one or more. Figure 7 A processor 401 is taken as an example. The processor 401, storage device 402, display screen 403, input device 404, output device 405 and communication device 406 of the electronic device can be connected via a bus or other means.

[0122] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions. When the instructions are executed on a computer, the computer executes the data stream processing method described in any one of the above embodiments.

[0123] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the data stream processing method described in any one of the above embodiments.

[0124] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0125] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments, and the embodiments can be referred to and combined with each other. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0126] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.< / version> < / version>

Claims

1. A data stream processing method, It is characterized in that Applied to a first computer room in a distribution system, the first computer room being any computer room in the distribution system, the method comprising: If an update instruction for a target data stream submitted by a first client is received, a computer room subscribing to the target data stream is determined according to the subscription relationship synchronized between the computer rooms in the distribution system, and the target computer room is obtained; the subscription relationship is used to indicate the data stream subscribed by the second client connected to each computer room through the computer room; Sending a data stream update message and the updated data of the target data stream to the target computer room; the data stream update message is used to instruct the target computer room to push the updated data to a third client connected to the target computer room; There are instances deployed in the first computer room, and each instance deployed in the first computer room is connected to at least one instance in another computer room; The sending of the data stream update message and the update data of the target data stream to the target computer room includes: The data stream update message and the update data are sent to the connected instances in the target computer room through each of the instances in the first computer room.

2. The method according to claim 1, It is characterized in that At least two instances are deployed in the first computer room, and the method further includes: Transmitting the updated data to each of the instances in the first computer room, so as to locally store the updated data in each of the instances; Re-establishing a connection between the first instance and a non-faulty instance in the first computer room in response to a connection request sent by the first instance in the second computer room; the connection request is sent by the first instance when a failure occurs in an instance connected to the first computer room; The updated data locally stored in the non-faulty instance is synchronized to the first instance based on the connection.

3. The method according to claim 2, It is characterized in that The transmitting the updated data to each of the instances in the first computer room so as to locally store the updated data in each of the instances includes: Setting version information for the update data; The update data and the version information are transmitted to each of the instances in the first computer room, and the update data is stored locally in the instance through the instance when the version information is higher than the version information currently held by the instance.

4. The method according to any one of claims 2 to 3, It is characterized in that The method further comprises: In the case of receiving a subscription request sent by the fourth client, generating a subscription relationship corresponding to the subscription request; the subscription relationship is used to represent the corresponding relationship between the first computer room and the data stream subscribed by the fourth client through the subscription request; The subscription relationship is synchronized to instances in any other computer room connected to the instance through each of the instances in the first computer room.

5. The method according to any one of claims 2 to 3, It is characterized in that The distribution system further includes a global module, wherein the global module stores relevant information of the instances deployed in each of the computer rooms; the method further includes: Acquire relevant information of the instances deployed in each of the other computer rooms from the global module; For any of the instances in the first computer room, a long connection is established between the instance and an instance in each of the other computer rooms according to relevant information of the instances deployed in each of the other computer rooms.

6. The method according to any one of claims 2 to 3, It is characterized in that The method further comprises: If a directory creation instruction is received from a second instance in another computer room, a receiving directory corresponding to the data stream directory indicated by the directory creation instruction is created through a third instance in the first computer room, and M marking files corresponding to the receiving directory are created; the receiving directory is used to store the relevant data of the data stream published under the data stream directory, the M number is the number of the second instance, and the third instance is connected to the second instance; When a deletion instruction is received from any of the second instances, one of the marked files is deleted through the third instance, and when all the marked files are deleted, the receiving directory is deleted.

7. A data stream processing device, It is characterized in that Applied to a first computer room in a distribution system, the first computer room being any computer room in the distribution system, the device comprising: A determination module, configured to determine, upon receiving an update instruction for a target data stream submitted by a first client, a computer room subscribed to the target data stream according to a subscription relationship synchronized between computer rooms in the distribution system, and obtain a target computer room; the subscription relationship is used to indicate a data stream subscribed by a second client connected to each computer room through the computer room; A sending module, used for sending a data stream update message and the update data of the target data stream to the target computer room; the data stream update message is used to instruct the target computer room to push the update data to a third client connected to the target computer room; There are instances deployed in the first computer room, and each instance deployed in the first computer room is connected to at least one instance in another computer room; Wherein, the sending module is also used for: The data stream update message and the update data are sent to the connected instances in the target computer room through each of the instances in the first computer room.

8. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

9. An electronic device, It is characterized in that include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the methods of claims 1 to 6 when executing the program.

Images