Live broadcast stream processing method, caching and analyzing server and computer equipment

Abstract

The invention discloses a live broadcast stream processing method, a cache server, an analysis server, computer equipment and a storage medium. The live broadcast stream processing method in one embodiment comprises the following steps: acquiring a live broadcast stream at a preset frame number interval; calculating the frame length of each frame in the live broadcast stream; calculating a frame length fluctuation average value of the live broadcast stream according to the frame length, wherein the frame length fluctuation average value is a frame length distribution mean square error of the live broadcast stream within a preset frame number interval; and when the frame length fluctuation average value of the live broadcast stream is greater than a preset sensitivity threshold, pushing a pull stream notification to an analysis server, so that the analysis server pulls the live broadcast stream within a preset frame number interval based on the pull stream notification to perform live broadcast stream analysis. According to the embodiment provided by the invention, the resource consumption can be reduced under the condition of ensuring the AI analysis efficiency of the live broadcast stream.

Description

Technical Field

[0001] The present disclosure relates to the field of communication technology, and in particular to a live stream processing method, a cache server, an analysis server, a computer device, and a storage medium. Background Art

[0002] In the live broadcast system, the artificial intelligence (AI) system requires huge CPU, GPU and memory resources when performing real-time image analysis of live streams. Especially when a large number of live streams are online, AI analysis is required at the same time, and the data transmission after analysis will occupy a lot of bandwidth. Existing equipment is limited by computing power and can only carry a limited number of routes. In actual applications, we can ask customers to expand the GPU and memory resources used for AI analysis as the number of live streams increases. Although this approach can solve the problem of increasing live streams without affecting the efficiency of AI analysis, the customer's equipment costs, equipment and data maintenance costs are continuing to grow. Summary of the invention

[0003] In order to solve at least one of the above problems, the first embodiment of the present disclosure provides a method for processing a live stream, which is applied to a cache server, comprising:

[0004] Get the live stream with preset frame interval;

[0005] Calculate the frame length of each frame in the live stream;

[0006] Calculate the average value of frame length fluctuation of the live stream based on the frame length;

[0007] When the frame length fluctuation average value of the live stream is greater than a preset sensitivity threshold, a stream pull notification is pushed to the analysis server, so that the analysis server pulls the live stream within a preset frame interval for live stream analysis based on the stream pull notification.

[0008] Optionally, before calculating the frame length fluctuation average value of the live stream according to the frame length, the method further includes:

[0009] Determine whether there is an initial frame length fluctuation average value;

[0010] When there is no initial frame length fluctuation average value, an initialization analysis is performed and a sensitivity value is calculated.

[0011] Optionally, calculating the frame length fluctuation average value of the live stream according to the frame length further includes:

[0012] Calculate the expected frame length distribution within a preset frame number interval;

[0013] Calculate the mean square error of the frame length distribution within the preset frame number interval.

[0014] Optionally, the frame length distribution of the live stream satisfies a Gaussian distribution.

[0015] Optionally, the live stream with a preset frame spacing includes at least two key frames, and / or

[0016] The same type of frame data in the live stream is used to calculate the frame length and the frame length fluctuation average value.

[0017] Optionally, the live stream is a live data stream with a variable bit rate.

[0018] Optionally, before obtaining the live stream with a preset frame number spacing, the method further includes: obtaining a range of a Gaussian distribution frame length fluctuation average value of the live stream with a preset frame number spacing in a static state and in a state of slight fluctuation allowed by the system,

[0019] The Gaussian distribution frame length fluctuation average is the mean square error of the Gaussian distribution satisfied by the live stream in a static state or in a system-allowable fluctuation state.

[0020] The preset sensitivity threshold is the upper limit of the range of the average value of the Gaussian distribution frame length fluctuation.

[0021] Optionally, after the stream pulling message has been sent, when the average value of the frame length fluctuation of the live stream is less than or equal to a preset sensitivity threshold, a stop stream pulling notification is pushed to the analysis server, so that the analysis server stops pulling the live stream based on the stop stream pulling notification, thereby stopping the analysis.

[0022] A second aspect of the present disclosure provides a method for processing a live stream, which is applied to an analysis server and includes:

[0023] Obtain the stream pull notification from the cache server. The stream pull notification is generated according to the processing method described above.

[0024] Based on the streaming service notification, the live stream within the preset frame interval is obtained from the external streaming end server and analyzed.

[0025] Optionally, the processing method further includes: after receiving the stream pulling message, if no new stream pulling message is received for more than a predetermined time, stopping the stream pulling analysis.

[0026] A third aspect of the present disclosure provides a cache server, including:

[0027] The communication component obtains the live stream with a preset frame interval;

[0028] A calculation component calculates the frame length of each frame in the live stream, and calculates the frame length fluctuation average value of the live stream based on the frame length, where the frame length fluctuation average value is the mean square error of the frame length distribution of the live stream within a preset frame number interval;

[0029] The message notification component pushes a stream pull notification to the analysis server when the mean square error of the frame length distribution is greater than a preset sensitivity threshold, so that the analysis server pulls the live stream within the preset frame interval based on the stream pull notification for live stream analysis.

[0030] A fourth aspect of the present disclosure provides an analysis server, comprising:

[0031] The communication component obtains a stream pull notification from the cache server, where the stream pull notification is generated according to the processing method described above;

[0032] The analysis component obtains the live stream within a preset frame interval from the external stream pulling end server based on the stream pulling service notification and performs analysis.

[0033] A fifth aspect of the present disclosure provides a computer-readable storage medium having a computer program stored thereon.

[0034] When the program is executed by a processor, the method according to the first aspect above is implemented, or

[0035] When the program is executed by a processor, the method described in the second aspect above is implemented.

[0036] A sixth aspect of the present disclosure provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor.

[0037] When the processor executes the program, the method described in the first aspect is implemented, or

[0038] When the processor executes the program, the method described in the second aspect is implemented.

[0039] The beneficial effects of the present disclosure are as follows:

[0040] In response to the existing problems, the present invention develops a live stream processing method, a cache server, an analysis server, a computer device and a storage medium. By calculating the frame length of each frame in a fixed number of frames and using the mean square error of the frame length distribution as a judgment basis for live stream analysis to determine whether to perform stream pull analysis, the resource consumption of the central processing unit and the graphics processing unit can be reduced while ensuring the efficiency of live stream AI analysis, the throughput of AI-related devices can be greatly improved, and the method has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 A flowchart showing a method for processing a live stream according to an embodiment of the present disclosure is shown;

[0043] Figure 2 A block diagram of the live stream processing system architecture is shown to which the live stream processing method according to an embodiment of the present disclosure can be applied;

[0044] Figure 3 A flowchart showing a method for processing a live stream according to an embodiment of the present disclosure is shown;

[0045] Figure 4 A diagram showing an interface configuration interface of a cache server in a live stream processing method according to another embodiment of the present disclosure;

[0046] Figure 5 A frame length statistics chart of a variable bit rate live stream in a static scene is shown;

[0047] Figure 6 A structural diagram of a cache server according to an embodiment of the present disclosure is shown;

[0048] Figure 7 A structural diagram of an analysis server according to an embodiment of the present disclosure is shown;

[0049] Figure 8 A schematic diagram of the structure of a computer device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0050] In order to more clearly illustrate the present disclosure, the present disclosure is further described below in conjunction with preferred embodiments and drawings. Similar components in the drawings are represented by the same reference numerals. It should be understood by those skilled in the art that the content described below is illustrative rather than restrictive, and should not be used to limit the scope of protection of the present disclosure.

[0051] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, similar words such as "one", "one" or "the" do not indicate quantitative restrictions, but indicate that there is at least one. "Include" or "comprising" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

[0052] Reference Figure 1 As shown, an embodiment of the present disclosure provides a method for processing a live stream, which is applied to a cache server, including:

[0053] Step S1, obtaining a live stream with a preset frame interval;

[0054] Step S2, calculating the frame length of each frame in the live stream;

[0055] Step S3, calculating the frame length fluctuation average value of the live stream according to the frame length;

[0056] Step S4: When the frame length fluctuation average value of the live stream is greater than a preset sensitivity threshold, a stream pull notification is pushed to the analysis server, so that the analysis server pulls the live stream within a preset frame interval for live stream analysis based on the stream pull notification.

[0057] In this embodiment, the frame length of each frame in a fixed number of frames is calculated, and the mean square error of the frame length distribution is used as a judgment basis for live stream analysis to determine whether to perform stream pull analysis. This can reduce the resource consumption of the central processing unit and graphics processing unit while ensuring the efficiency of live stream AI analysis, and greatly improve the throughput of AI-related devices.

[0058] In order to facilitate understanding of the technical solution of this embodiment, the following first combines Figure 2 The architecture diagram of a live streaming processing system to which the above method provided in this embodiment can be applied is described.

[0059] like Figure 2 As shown, the live streaming processing system includes: a collection end 1, a streaming media service end 2 and a playback end 3.

[0060] The acquisition end 1 is the source end for providing the live stream, and may include: at least one of a camera, a proxy streaming tool, a third-party streaming tool, and other tools for providing the live stream. The camera may be a high-definition camera, and the camera may be directly connected to the streaming media server 2 and push the live stream. Some cameras have the function of registering with the streaming media server 2, and can actively push the live stream directly to the streaming media server 2. In other cases, the camera can push the live stream it collects to the streaming media server 20 with the help of a streaming tool. In this case, the streaming tool may be a proxy streaming tool, a third-party streaming tool, or other tools. These tools may all be tools that connect to the camera and the streaming media server, and the third-party streaming tool may be, for example, FFMPEG, which is an open source multimedia processing tool, and the proxy streaming tool may be, for example, an open source push-pull stream service or other push-pull stream plug-in. These tools push the live stream collected by the camera to the streaming media server 2 by configuring the address of the camera. Of course, the way in which the acquisition end 1 provides the live stream is not limited to this, and other ways of obtaining and pushing the live stream are also protected in the embodiments of the present disclosure.

[0061] The streaming media service end 2 may include: a push stream end server 201, a cache server 202, a pull stream end server 203, an analysis server 204, a storage server 205, a configuration server 206, and an ETCD server 207. Among them, the pull stream end server 203 includes a first pull stream end server 203-1 and a second pull stream end server 203-2. When not distinguished, the two are collectively referred to as the push and pull stream end server 203. Each of the above servers can be a server cluster to provide higher computing power and processing capabilities. For example, the push stream end server 201 can be a push stream end server cluster, the pull stream end server 203 can be a pull stream end server cluster, and the analysis server 204 can be an analysis server cluster.

[0062] Specifically, the push stream server 201 is responsible for receiving the live stream of the standard protocol. These live streams are, for example, live streams from cameras, or tools such as FFMPEG. The push stream server 201 can obtain the live stream in a push stream mode or a pull stream mode. For example, it can obtain the live stream actively pushed by the acquisition end 1, or actively pull the live stream through the address configured by the acquisition end 1. The push stream server 201 pushes the received live stream to the cache server 202. Correspondingly, the pull stream server 203 is responsible for providing live broadcast services to the playback end 3. The cache server 202 caches the live stream and provides value-added services of the live stream, such as screenshots, watermarks, transcoding, message notifications, image fluctuation recognition, etc. The first pull stream server 203-1 performs ① message subscription on the cache server 202, ② receiving the new video stream source information pushed by the cache server 202, ③ establishing a new connection with the cache server 202 to request the video stream, and ④ receiving the video or audio stream packet pushed by the cache server 202. The storage server 205 is used for storing the live streaming data on disk. Specifically, it pulls the cached live streaming from the cache server 202 for storage. The second stream pulling end server 203-2 orders the live streaming from the storage server 205 and forwards it to the playback end 3. In addition, the streaming media server 2 may also include a configuration server 206 and an ETCD server 207. The configuration server 206 is responsible for the unified management of the configuration items of each server in the configuration system, the status monitoring of various servers, and the online monitoring of various streams. The ETCD server 207 is used for the registration, discovery and configuration cache of each server.

[0063] The playback end 3 can be a device with playback function, which can be collectively referred to as a player. The stream pulling end server 203 pulls the live stream from the cache server or storage server. The address of the stream pulling end server 203 is stored in the playback end 3. After the stream pulling end server 203 obtains the live stream, the playback end 3 can obtain the live stream and play it.

[0064] The following is based on Figure 2 The system architecture and Figure 3A flowchart is provided to describe in detail the method for processing the live stream according to an embodiment of the present disclosure.

[0065] For example, the processing method described below takes the live monitoring of a park as an application scenario. The park can be, for example, a tourist area, a smart park, or a block. In such a scenario, a large number of cameras are often deployed and fixed in position, and the occurrence rate of events is low. Of course, the application scenario is not limited to this. As long as there are scenes with a large number of static images during the live stream acquisition process, the processing method disclosed in this disclosure can achieve good results.

[0066] In step S1, a live stream with a preset frame interval is obtained.

[0067] The live stream may be a stream using the method described above. Figure 2 The one or more cameras in the acquisition terminal 1 in the embodiment collect the live stream, and provide the live stream to the streaming server 201 by directly connecting to the streaming server 201 or indirectly connecting to the streaming tool. The streaming server 201 pushes the live stream to the cache server 202, so that the cache server 202 obtains the live stream.

[0068] In specific implementation, the cache server can cache multiple live streams collected by multiple cameras at the same time, and the interface identifies different live streams through live stream IDs. Figure 4 As shown, the figure shows an example of a REST FUL interface interface diagram of a cache server 202, and the cache server 202 uses this interface to send configuration. Figure 4 It can be seen that in this example, the cache server 202 provides the / live / gaussian / {sid} interface, which corresponds to the POST, GET, and DELETE methods. For example, POST is used to set the resource optimization detection mechanism corresponding to the live stream ID, GET is used to obtain the configuration information of the corresponding live stream ID, and DELETE is used to delete the configuration that is effective for the corresponding live stream ID.

[0069] During the subsequent operation, the cache server 202 parses each parameter in the background. If able=true, the service is started, otherwise it is closed; if png=true, the statistical information of the frame length is stored as a PNG image for user analysis; if check=true, the calculation of the effective range of the frame length fluctuation average is started; step represents the statistical length (the number of frames calculated at one time), and the relevant functions will be described in detail in the subsequent steps and will not be repeated here.

[0070] Preferably, the live stream is a live data stream with a variable bit rate. When a live data stream with a variable bit rate is used, the bit rate change is more obvious than the fluctuation change of the live data stream with a fixed bit rate, so that the screening effect of the live stream is better.

[0071] In particular, in an embodiment of the present disclosure, the cache server 202 processes a live stream with a preset frame spacing. Preferably, the live stream with the preset frame spacing should include at least two key frames, and a key frame refers to a frame containing complete picture information. Only when the live stream with the preset frame spacing used for subsequent calculations includes at least two key frames can the fluctuating change state of the live stream be accurately reflected. It can be understood that when the number of frames in the preset frame spacing is too small to include key frames. Exemplarily, the preset frame spacing can be 50 frames. Of course, if the computing power permits, a larger value can also be selected.

[0072] In step S2, the frame length of each frame in the live stream is calculated.

[0073] It should be noted that the live stream in this step refers to the live stream with the preset frame number spacing described above. For example, if the preset frame number spacing is 50 frames, this step calculates the frame length of each frame in the live stream of 50 frames.

[0074] In the embodiments of the present disclosure, the frame length refers to the length of the video data carried by a complete frame. Due to the different streaming protocols, there are certain differences in the calculation of frame length. For example, the RTMP protocol can directly calculate the length of a frame, while the RTSP protocol or the RTP protocol needs to collect the complete data of a frame first (to determine whether data is lost) and then calculate the frame length. If data loss is found during the statistical process, for example, if the packet needs to be discontinuous, the frame data cannot be used for subsequent calculations.

[0075] Optionally, the cache server 202 obtains and counts the frame length data in a queue cache manner. The frame length data obtained by statistics can also be processed into X and Y axis data required for generating the PNG image, where the X axis can be time (in milliseconds) and the Y axis represents the frame length (in bits), so as to facilitate direct application of subsequent services.

[0076] Preferably, in the embodiment of the present disclosure, independent thread A is started for asynchronous caching and subsequent frame length fluctuation average calculation, independent thread B is used to generate real-time PNG images, and independent thread C is used to compare the frame length fluctuation average with a preset sensitivity threshold, thereby improving the concurrency capability of the system. In the embodiment of the present disclosure, the frame length fluctuation average may also be referred to as a "sensitivity value" to reflect the measurement of the sensitivity to frame length fluctuation in the embodiment of the present disclosure.

[0077] Specifically, in this step, after thread A is started, it waits to receive the frame length and timestamp of the live stream, and sets the value of "step" to the preset frame spacing, for example, step = 50, which determines the upper limit of the cache. If the upper limit is not reached, the business logic is directly launched. When each data frame of the live stream arrives, the data length L it carries is calculated and stored in the cache, and the cache is pushed to thread B for generating PNG.

[0078] When using queues for caching, the order of frames also needs to be guaranteed. In addition, cache server 202 can also use message queues to temporarily cache messages of different live streams that need to be distributed. When using message queues to cache messages, it can use asynchronous processing to improve the concurrent capability of the system.

[0079] In addition, for live data streams, no matter what encoding method is used, they usually contain I frames, because I frames are independently encoded pictures that contain complete picture information, and key frames usually refer to I frames. In some encoding methods, such as IPB encoding, in addition to I frame data, P frame data and B frame data are also required. In this case, in order to ensure the consistency of the frame length standard used to evaluate the frame length fluctuation average value obtained each time, the same type of frame data in the live stream is used for frame length calculation and subsequent frame length fluctuation average value calculation. For example, for all live streams from a collection source, I frames are used to calculate the frame length each time, or for all live streams from a collection source, I frames and P frames are used to calculate the frame length each time, or for all live streams from a collection source, I frames, P frames and B frames are used to calculate the frame length each time.

[0080] After step S2, refer to Figure 3 As shown, the processing method also includes: determining whether there is an initial frame length fluctuation average value; when there is no initial frame length fluctuation average value, performing initialization analysis and calculating the sensitivity value.

[0081] Specifically, for a live stream of a source, an object is created for the live stream, and the object contains an initial frame length fluctuation average value contained in a field identifier. The value of the initialization action at the beginning of the processing is "0", that is, it is empty. In the embodiment of the present disclosure, the process of setting the processing includes an initialization analysis phase. If the bit is judged to be "0", it means that there is no frame length fluctuation average value in the processing flow, and it needs to be assigned a value before the subsequent steps can be performed.

[0082] Reference Figure 3 As shown, when it is determined that there is no initial frame length fluctuation average value, it is necessary to perform initialization analysis and calculate the frame length fluctuation average value.

[0083] Specifically, for the first judgment step after the processing flow of a live stream from a source begins, the frame length fluctuation average value calculated by the initial analysis is the frame length fluctuation average value of the 50 frames received at this time. As can be seen from the following description, the frame length fluctuation average value represents the mean square error of the frame length distribution of the live stream of the 50 frames. For non-first judgment steps, the initial frame length fluctuation average value should be the frame length fluctuation average value calculated for the last 50 frames of the live stream.

[0084] In other words, in the embodiment of the present disclosure, step S3 can only be carried out when the initial frame length fluctuation average value is not empty. If it is not the first judgment step, because the previous frame length fluctuation average value has been stored at this position, a "yes" result can be obtained in the step of judging whether the frame length fluctuation average value exists, thereby entering the next step to calculate the frame length fluctuation average value of the live stream with the preset frame number spacing currently obtained, as shown below for details.

[0085] In step S3, the frame length fluctuation average value of the live stream is calculated based on the frame length.

[0086] Optionally, the frame length fluctuation average value is the mean square error of the frame length distribution of the live stream within a preset frame number interval. Specifically, the frame length distribution expectation within the preset frame number interval is calculated. The frame length distribution expectation satisfies:

[0087] E(X)=X1×p(X1)+X2×p(X2)+…+Xa×p(Xa)+…+Xn×p(Xn).

[0088] Wherein, n is the number of frames within the preset frame spacing. When the preset frame spacing is 50 frames, n is 50. Xa represents the frame length of the ath frame from the 1st to the nth frames. p(Xa) represents the probability of fluctuation in the ath frame. The frame length distribution is expected to represent the overall numerical average value of the deviation within the preset frame spacing.

[0089] After that, the mean square error of the frame length distribution within the preset frame number interval is calculated. The mean square error of the frame length distribution satisfies:

[0090]

[0091] The mean square error indicates the degree of deviation of the frame length of each frame in a preset frame interval, such as a 50-frame live stream, from the overall numerical average value of the offset.

[0092] Optionally, the distribution of the live stream follows a Gaussian distribution, and the mean square deviation of its frame length distribution can satisfy the calculation method of the mean square deviation of the Gaussian distribution.

[0093] By adopting Gaussian distribution, it is possible to more realistically simulate the fluctuation pattern of frame length in the live stream when fluctuations occur, thereby making the basis for analysis and judgment more accurate.

[0094] Specifically, the probability density formula of Gaussian distribution is:

[0095] Among them, the probability density representation is subject to the parameters μ, σ 2 Gaussian distribution, denoted as X~N(μ,σ 2 ). μ is the expectation of Gaussian distribution, and σ is the mean square error of Gaussian distribution. Among them, μ determines the position of Gaussian distribution, and σ determines the degree of dispersion of Gaussian distribution.

[0096] In the specific implementation, in thread A, when the calculated value of "step" reaches the upper limit, such as 50, it is necessary to calculate the expectation and mean square error of the Gaussian distribution. For example, when the 51st and subsequent frames arrive, first calculate the expectation of the cached 50 frames as μ, then calculate the mean square error σ, and push the mean square error to thread C as the average value of frame length fluctuation.

[0097] In an embodiment of the present disclosure, by utilizing the distribution law of the live stream and specifically calculating the mean square error that can characterize the degree of deviation of the live stream, and using it as the average value of the frame length fluctuation, it can be used to determine the degree of fluctuation of the live stream with a preset frame spacing currently obtained.

[0098] In step S4, when the frame length fluctuation average value of the live stream is greater than a preset sensitivity threshold, a stream pull notification is pushed to the analysis server, so that the analysis server pulls the live stream within a preset frame interval for live stream analysis based on the stream pull notification.

[0099] In this step, it should be firstly clarified that, in the embodiment of the present disclosure, a preset sensitivity threshold is pre-stored in the cache server 202. The process of obtaining the preset sensitivity threshold may be executed in the cache server 202.

[0100] Optionally, the live stream processing method disclosed in the present invention also includes: obtaining the range of the Gaussian distribution frame length fluctuation average value of the live stream with a preset frame number spacing in a static state and in a system-allowable fluctuation state, wherein the Gaussian distribution frame length fluctuation average value is the mean square deviation of the Gaussian distribution satisfied by the live stream in a static state or in a system-allowable fluctuation state, and the preset sensitivity threshold is the upper limit value of the range of the Gaussian distribution frame length fluctuation average value.

[0101] Specifically, refer to Figure 5 As shown, Figure 5 The figure shows a frame length statistics chart of a variable bit rate live stream in a static state, with the horizontal axis representing the timestamp in milliseconds (ms) and the vertical axis representing the frame length in bytes. Figure 5 It can be seen that in static scenes, the morphology is relatively regular, and the length of each frame is slightly different, which follows the Gaussian distribution (ie, normal distribution).

[0102] The inventors have found through probability statistics that, except in the static state, the fluctuation range of the frame length statistics of the live stream in the fluctuation state allowed by the system is also very small and also follows the Gaussian distribution. In the embodiments of the present disclosure, the fluctuation state allowed by the system refers to the occurrence of fluctuation events that the user does not pay attention to. The fluctuation state allowed by the system can be, for example, an insect flying within the imaging field of view of the camera, a pedestrian walking within the imaging field of view of the camera, or fallen leaves or branches blown by the wind within the imaging field of the camera, etc., and the results are not events that the park monitoring is concerned about.

[0103] It should be noted that the probability density formula applied when calculating the range of the average value of the Gaussian distribution frame length of the live stream in a static state or a system-allowable fluctuation state, as well as the calculation method of the expectation and mean square error are the same as above and will not be repeated here.

[0104] In the embodiments of the present disclosure, it is necessary to use a statistical method to determine the Gaussian distribution function satisfied by the live stream under static conditions and in a state of slight fluctuation allowed by the system, and obtain the Gaussian distribution frame length fluctuation average value within a preset frame number interval (for example, 50 frames) under the two distributions according to the respectively determined Gaussian distribution functions. The Gaussian distribution sensitivity value is the mean square error of the Gaussian distribution satisfied by the live stream in a static state and the mean square error of the Gaussian distribution satisfied by the live stream in a state allowed by the system. The range of the Gaussian distribution frame length fluctuation average value is the numerical interval formed by the Gaussian frame length fluctuation average value in a static state to the Gaussian frame length fluctuation average value in a state of maximum fluctuation allowed by the system.

[0105] In the embodiment of the present disclosure, the preset sensitivity threshold is used as a basis for determining whether the live stream acquired later is a fluctuation event that needs to be analyzed. Therefore, the preset sensitivity threshold is the upper limit of the range of the average value of the Gaussian distribution frame length fluctuation, that is, the average value of the Gaussian frame length fluctuation of the maximum fluctuation state allowed by the system.

[0106] It should be noted that the upper limit of this numerical range depends on the sensitivity setting of the application scenario of the system to fluctuations. As mentioned above, if the application scenario of this example is the monitoring of a park, then fluctuations such as people passing by can be ignored, so the upper limit of the Gaussian frame length fluctuation average value can be the mean square error of the Gaussian distribution that includes fluctuation events such as people passing by, and the preset frame length fluctuation average value is the upper limit. If the sensitivity requirements for environmental analysis are relatively high in some application scenarios, the preset frame length fluctuation average value can be determined according to the Gaussian distribution with smaller fluctuations. In other words, the preset sensitivity threshold in this article can be set as needed.

[0107] When implementing it specifically, Figure 4When the configuration item check in is "true", the cache server 202 will temporarily cache the calculated Gaussian frame length fluctuation average value. When the cache server 202 starts for a certain period of time and the screen is stable, for example, after about 30 seconds, it will count the frequency of occurrence of each Gaussian frame length fluctuation average value, and count the maximum and minimum Gaussian frame length fluctuation average values ​​to form the effective range of the frame length fluctuation average value, such as (0.00026~0.00030). The user can view this value range through the interface. Generally, the user configures the upper limit of the value range (0.00030) as the base point. If necessary for business, the cache server 202 will check this value range every once in a while, for example, every hour.

[0108] Of course, the server that performs the above process of obtaining the preset sensitivity threshold may also be other cloud servers. In the cache server 202, it is only necessary to obtain and set the preset sensitivity threshold in advance, which will not be elaborated here.

[0109] Returning to step S4 , the message notification of the cache server 202 is triggered outside the range of the above-mentioned preset sensitivity threshold.

[0110] Specifically, continue to refer to Figure 3 As shown, the currently calculated frame length fluctuation average value of the live stream is compared with the preset sensitivity threshold. When the frame length fluctuation average value is greater than the preset sensitivity threshold, it indicates that a fluctuation event that needs to be analyzed has occurred in the live stream of the current frame interval, indicating that the data of the live stream needs to be analyzed and cannot be ignored, and thus a stream pulling notification needs to be pushed to the analysis server 204. When the analysis server 204 receives the stream pulling notification from the cache server 202, it pulls the live stream from the first stream pulling end server 203-1 and analyzes the live stream.

[0111] In this step, as long as the calculated frame length fluctuation average value is outside the range of the preset sensitivity threshold, that is, greater than the preset frame length fluctuation average value, the message notification mechanism is automatically triggered. The structure of the message can be as follows:

[0112]

[0113] Among them, ID represents the message ID, URL represents the streaming address, Timeout represents the streaming timeout, and Timestamp represents the timestamp (in seconds). When the same stream is pushed multiple times, if the analysis server 204 has already pulled the stream, only Timeout needs to be updated. When calculating whether the same stream is within the average value of the frame length fluctuation, in principle, the message is pushed once every 1 second. This may cause processing problems in various scenarios so that the message can be pushed accurately.

[0114] It should be noted that, in the embodiments of the present invention, because the live stream is used as video data, when the queue cache method is used to cache and calculate the live stream with a preset frame interval, because of its queue stacking and popping method, each acquired live stream with a preset frame interval is also continuous. That is to say, it is not the first time to acquire, calculate and judge the live stream from the 1st to the 50th frames, and the second time to acquire, calculate and judge the live stream from the 51st to the 100th frames, but it should be the first time to acquire, calculate and judge the live stream from the 1st to the 50th frames, and the second time to acquire, calculate and judge the live stream from the 2nd to the 51st frames.

[0115] Taking into account that when fluctuations occur, they occur continuously, frame after frame, and a queue stack method is used to calculate and judge the above-mentioned frame length fluctuation average value, in order to avoid message redundancy, a stop streaming message can be set. After the streaming message has been sent, when the frame length fluctuation average value of the live stream is less than or equal to a preset sensitivity threshold, a stop streaming notification is pushed to the analysis server, so that the analysis server stops pulling the live stream based on the stop streaming notification, thereby stopping the analysis.

[0116] In this way, the cache server 202 can avoid redundant actions of the message sending mechanism and simplify the processing flow of the device.

[0117] The above settings can greatly optimize the analysis process of the analysis server 204 and improve efficiency, and can greatly improve the computing power of scenarios that require real-time large-scale analysis and detection of live streams.

[0118] Continue to combine Figure 2 The second aspect of the present disclosure provides another method for processing a live stream, which is applied to an analysis server and includes:

[0119] Obtain the stream pull notification from the cache server. The stream pull notification is generated according to the processing method described above.

[0120] Based on the streaming service notification, the live stream within the preset frame interval is obtained from the external streaming end server and analyzed.

[0121] Through the above settings, the analysis server can obtain and analyze the live stream within a preset time interval from the external streaming server only when it receives the streaming notification that needs to be analyzed from the cache server. This can significantly save the computing power of the analysis server, improve the application cost of the analysis server, and reduce the equipment cost under the same conditions.

[0122] Optionally, the processing method further includes: after receiving the stream pulling message, if no new stream pulling message is received for more than a predetermined time, stopping the stream pulling analysis.

[0123] This setting can further simplify the analysis operation process of the analysis server and avoid process loss caused by high-frequency message reception.

[0124] Combination Figure 2 and Figure 6 As shown, in response to the live stream processing method of the first aspect, an embodiment of the present disclosure further provides a cache server, including:

[0125] The communication component 211 obtains a live stream with a preset frame interval;

[0126] A calculation component 222 calculates the frame length of each frame in the live stream, and calculates the frame length fluctuation average value of the live stream according to the frame length, wherein the frame length fluctuation average value is the mean square error of the frame length distribution of the live stream within the preset frame number interval;

[0127] The message notification component 232 pushes a stream pulling notification to the analysis server when the mean square error of the frame length distribution is greater than a preset sensitivity threshold, so that the analysis server pulls the live stream within the preset frame number interval based on the stream pulling notification for live stream analysis.

[0128] The above settings can greatly optimize the analysis process of the analysis server 204 and improve efficiency, and can greatly improve the computing power of scenarios that require real-time large-scale analysis and detection of live streams.

[0129] Combination Figure 2 and Figure 7 As shown, in response to the live stream processing method of the first aspect, an embodiment of the present disclosure further provides an analysis server, including:

[0130] The communication component 214 obtains a stream pull notification from the cache server, where the stream pull notification is generated according to the processing method described in the second aspect above;

[0131] The analysis component 224 obtains the live stream within a preset frame interval from the external stream pulling end server based on the stream pulling service notification and performs analysis.

[0132] With the above settings, the analysis server can obtain and analyze the live stream within a preset time interval from the external stream pulling server only when it receives the stream pulling notification that needs to be analyzed from the cache server. This can significantly save the computing power of the analysis server, improve the application cost of the analysis server, and reduce the equipment cost under the same conditions.

[0133] Another embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements: obtaining a live stream with a preset frame number spacing; calculating the frame length of each frame in the live stream; calculating the frame length fluctuation average value of the live stream according to the frame length, the frame length fluctuation average value being the mean square error of the frame length distribution of the live stream within the preset frame number spacing; when the frame length fluctuation average value of the live stream is greater than a preset sensitivity threshold, pushing a stream pull notification to an analysis server, so that the analysis server pulls the live stream within the preset frame number spacing based on the stream pull notification to perform live stream analysis.

[0134] Another embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements: obtaining a stream pulling notification from a cache server, wherein the stream pulling notification is generated according to the processing method described in the first aspect above, and based on the stream pulling service notification, obtaining a live stream within a preset frame interval from an external stream pulling end server and analyzing the live stream.

[0135] In practical applications, the computer-readable storage medium can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0136] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0137] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0138] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0139] like Figure 8 As shown, another embodiment of the present disclosure provides a structural schematic diagram of a computer device. Figure 8 The computer device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0140] like Figure 8 As shown, the computer device 12 is in the form of a general-purpose computing device. The components of the computer device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components (including the system memory 28 and the processing unit 16).

[0141] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0142] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0143] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 8 not shown, usually called a "hard drive"). Although Figure 8 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present disclosure.

[0144] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in the memory 28, such program modules 42 including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described in the present disclosure.

[0145] The computer device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), one or more devices that enable a user to interact with the computer device 12, and / or any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed through an input / output (I / O) interface 22. Furthermore, the computer device 12 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter 20. Figure 8 As shown, the network adapter 20 communicates with other modules of the computer device 12 via the bus 18. It should be understood that although Figure 8 Not shown, other hardware and / or software modules may be used in conjunction with computer device 12, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0146] The processor unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing a live streaming processing method provided in an embodiment of the present disclosure.

[0147] In response to the existing problems, the present invention develops a live stream processing method, a cache server, an analysis server, a computer device and a storage medium. By calculating the frame length of each frame in a fixed number of frames and using the mean square error of the frame length distribution as a judgment basis for live stream analysis to determine whether to perform stream pull analysis, the resource consumption of the central processing unit and the graphics processing unit can be reduced while ensuring the efficiency of live stream AI analysis, the throughput of AI-related devices can be greatly improved, and the method has broad application prospects.

[0148] Obviously, the above embodiments of the present disclosure are merely examples for clearly illustrating the present disclosure, and are not limitations on the implementation methods of the present disclosure. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made on the basis of the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present disclosure are still within the protection scope of the present disclosure.

Claims

1. A method for processing a live stream, applied to a cache server, characterized in that: include: Get the live stream with preset frame interval; Calculating the frame length of each frame in the live stream; Calculate the frame length fluctuation average value of the live stream according to the frame length; When the frame length fluctuation average value of the live stream is greater than a preset sensitivity threshold, a stream pulling notification is pushed to the analysis server, so that the analysis server pulls the live stream within the preset frame number interval for live stream analysis based on the stream pulling notification.

2. The processing method according to claim 1, characterized in that: The frame length fluctuation average value is: the mean square error of the frame length distribution of the live stream within the preset frame number interval.

3. The processing method according to claim 1, characterized in that: Before calculating the frame length fluctuation average value of the live stream according to the frame length, the method further includes: Determine whether there is an initial frame length fluctuation average value; When the initial frame length fluctuation average value does not exist, an initialization analysis is performed and the frame length fluctuation average value is calculated.

4. The processing method according to claim 1, characterized in that: The calculating the frame length fluctuation average value of the live stream according to the frame length further comprises: Calculating the expected frame length distribution within the preset frame number interval; Calculate the mean square error of the frame length distribution within the preset frame number interval.

5. The processing method according to claim 4, characterized in that: The frame length distribution of the live stream satisfies the Gaussian distribution.

6. The processing method according to claim 1, characterized in that: The live stream with the preset frame spacing includes at least two key frames, and / or The frame data of the same type in the live stream is used to calculate the frame length and the frame length fluctuation average value.

7. The processing method according to claim 1, characterized in that: The live stream is a live data stream with a variable bit rate.

8. The processing method according to claim 1, characterized in that: Before obtaining the live stream with a preset frame spacing, the method further includes: Obtain the range of the average value of the Gaussian distribution frame length fluctuation of the live stream with the preset frame number spacing in a static state and in a fluctuation state allowed by the system, The Gaussian distribution frame length fluctuation average value is the mean square error of the Gaussian distribution satisfied by the live stream in a static state or in a fluctuation state allowed by the system. The preset sensitivity threshold is the upper limit of the range of the average value of the Gaussian distribution frame length fluctuation.

9. The processing method according to claim 1, characterized in that: After the stream pulling message has been sent, when the frame length fluctuation average value of the live stream is less than or equal to a preset sensitivity threshold, a stop stream pulling notification is pushed to the analysis server, so that the analysis server stops pulling the live stream based on the stop stream pulling notification, thereby stopping the analysis.

10. A method for processing a live stream, applied to an analysis server, characterized in that: include: Obtain a stream pull notification from a cache server, wherein the stream pull notification is generated by the processing method according to any one of claims 1 to 9, Based on the stream pulling service notification, a live stream within a preset frame interval is obtained from an external stream pulling end server and analyzed.

11. The processing method according to claim 10, characterized in that: Also includes: After receiving the stream pulling message, if no new stream pulling message is received within a predetermined time, the stream pulling analysis is stopped.

12. A cache server, characterized in that: include: The communication component obtains the live stream with a preset frame interval; A calculation component calculates the frame length of each frame in the live stream, and calculates the frame length fluctuation average value of the live stream according to the frame length, wherein the frame length fluctuation average value is the mean square error of the frame length distribution of the live stream within the preset frame number interval; The message notification component pushes a stream pulling notification to the analysis server when the mean square error of the frame length distribution is greater than a preset sensitivity threshold, so that the analysis server pulls the live stream within the preset frame number interval based on the stream pulling notification for live stream analysis.

13. An analysis server, characterized in that: include: A communication component obtains a stream pull notification from a cache server, wherein the stream pull notification is generated by the processing method according to any one of claims 1 to 9; The analysis component obtains the live stream within a preset frame interval from the external stream pulling end server based on the stream pulling service notification and performs analysis.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 9 is implemented, or When the program is executed by a processor, the method according to claim 10 or 11 is implemented.

15. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 9 is implemented, or When the processor executes the program, the method according to claim 10 or 11 is implemented.

Images