Live Client Freezing Detection Method and System

Abstract

The present application discloses a method for detecting lags in a live broadcast client, which is applied to a server. The server is communicatively connected to a client and a test end through a local area network. The method includes: obtaining a video stream pushed by the client through the local area network, where the video stream is a playback screen of a test video played by the test end captured by the client, and a pre-set clock is superimposed and displayed in the playback screen; extracting a time series from the video stream, where the time series is time information on the clock; determining whether the video stream has lags according to whether the time series increases in sequence; saving the determination result and displaying it on the front end. The present application also discloses a system for detecting lags in a live broadcast client, an electronic device, and a computer-readable storage medium. Thus, it is possible to automatically detect the live broadcast lag situation caused by the client, with accurate detection results, comprehensive coverage, and the interference of the online streaming media server to the detection results can be excluded.

Description

Technical Field

[0001] The present application relates to the field of live broadcast technology, and in particular, to a method, a system, an electronic device, and a computer-readable storage medium for detecting lags in a live broadcast client. Background Art

[0002] With the development of computer technology and mobile terminal applications, video live broadcast has become a trend, and more and more users watch and participate in live broadcast scenarios through mobile terminals. The host uses a live broadcast tool at the host side to push the video stream, and after the audience enters the live broadcast room, they receive the video stream pushed by the host and watch it. If there are problems at the host side, the network, or the audience side, it may cause lags in the live broadcast.

[0003] Currently, the detection solutions for live broadcast lags mainly include manual detection, third-party detection SDKs (Software Development Kits), data burying points, data stream monitoring, etc. However, these methods may have defects such as consuming a lot of manpower, low accuracy, and low data security. Especially when it is necessary to detect the lags caused by the live broadcast APP (Application) on the client side, these detection methods cannot exclude the lags caused by the poor network stability of the online streaming media server. Summary of the Invention

[0004] The main purpose of the present application is to propose a method, a system, an electronic device, and a computer-readable storage medium for detecting lags in a live broadcast client, aiming to solve the problem of how to conveniently and accurately detect the lags in the live broadcast caused by the live broadcast APP on the client side.

[0005] To achieve the above object, an embodiment of the present application provides a method for detecting lags in a live broadcast client, which is applied to a server. The server is communicatively connected to a client and a test end through a local area network. The method includes:

[0006] Obtain a video stream pushed by the client through the local area network. The video stream is the playback picture of the test video played by the test end captured by the client, and a pre-set clock is superimposed and displayed in the playback picture;

[0007] Extract a time series from the video stream. The time series is the time information on the clock;

[0008] Determine whether there is a lag in the video stream according to whether the time series increases in sequence.

[0009] Optionally, the clock is a millisecond-level clock displayed at any predetermined position on the display screen of the test end.

[0010] Optionally, the clock includes a plurality of clocks displayed at a plurality of predetermined positions on the display screen at the test end, and the times displayed by the plurality of clocks are kept consistent.

[0011] Optionally, the clock includes two millisecond-level clocks displayed above and below the left edge of the display screen at the test end.

[0012] Optionally, the obtaining of the video stream pushed by the client through the local area network includes:

[0013] Receiving a push stream request sent by the client in the local area network and returning a push stream address to the client;

[0014] Obtaining the video stream pushed by the client according to the push stream address.

[0015] Optionally, the extracting of the time series from the video stream includes:

[0016] Using optical character recognition technology to recognize the time information of the clock from each frame of video image of the video stream to obtain the time series.

[0017] Optionally, the extracting of the time series from the video stream includes:

[0018] Saving a video recording of a segment of the video stream at a predetermined interval;

[0019] Using optical character recognition technology to recognize the time information on the clock from each frame of video image of the video recording;

[0020] Sorting the recognized time information in the playing order of the video recording to obtain the time series.

[0021] Optionally, the extracting of the time series from the video stream includes:

[0022] Cutting each frame of video image into two clock regions;

[0023] Respectively using optical character recognition technology to recognize the time information on the clock for the two clock regions to obtain two recognition results;

[0024] Selecting the one with the highest confidence from the two recognition results as the time information recognized for this frame of video image.

[0025] Optionally, the determining of the video stream for jitter based on whether the time series increases in sequence includes:

[0026] In the case where the difference in time information between this frame of video image and the previous frame of video image is greater than a preset value, it is determined that the video stream has jitter.

[0027] Optionally, the method further includes:

[0028] Saving the determination result, the test video, the video recording, and the time series for review, and presenting the determination result at the front end.

[0029] Optionally, the method further includes:

[0030] Running the processes of obtaining the video stream, extracting the time series, and performing stuttering determination respectively using multiple processes simultaneously.

[0031] In addition, to achieve the above object, an embodiment of the present application further provides a live client stuttering detection system, where the system includes:

[0032] An obtaining module, configured to obtain a video stream pushed by a client through a local area network, where the video stream is a playback picture of a test video played by a test end captured by the client, and a pre-set clock is superimposed and displayed in the playback picture;

[0033] An extraction module, configured to extract a time series from the video stream, where the time series is time information on the clock;

[0034] A determination module, configured to perform stuttering determination on the video stream according to whether the time series increases in sequence.

[0035] To achieve the above object, an embodiment of the present application further provides an electronic device, where the electronic device includes: a memory, a processor, and a live client stuttering detection program stored on the memory and executable on the processor, and when the live client stuttering detection program is executed by the processor, the live client stuttering detection method as described above is implemented.

[0036] To achieve the above object, an embodiment of the present application further provides a computer-readable storage medium, where a live client stuttering detection program is stored on the computer-readable storage medium, and when the live client stuttering detection program is executed by a processor, the live client stuttering detection method as described above is implemented.

[0037] The live client stuttering detection method, system, electronic device, and computer-readable storage medium provided by the embodiments of the present application can form a local area network, superimpose and display a clock in the picture of playing a test video at the test end, the client captures the playback picture of the test end through a live broadcast APP and pushes the stream to the server through the local area network, the server extracts the time series in the live video stream using OCR technology, thereby automatically detecting the live stuttering situation caused by the client, with accurate detection results, comprehensive coverage, and the interference of the online streaming media server to the detection results can be excluded, which is convenient for problem positioning. Description of the Drawings

[0038] Figure 1 An application environment architecture diagram for implementing various embodiments of the present application;

[0039] Figure 2 Another form of application environment architecture diagram for implementing various embodiments of the present application;

[0040] Figure 3 A flowchart of a method for detecting lags in a live broadcast client proposed in the first embodiment of the present application;

[0041] Figure 4 A schematic diagram of ghosting and afterimage pictures in the present application;

[0042] Figure 5 A schematic diagram of the experimental results of a comparative study on a clock display method in the present application;

[0043] Figure 6 A schematic diagram of the hardware architecture of an electronic device proposed in the second embodiment of the present application;

[0044] Figure 7 A schematic diagram of the modules of a system for detecting lags in a live broadcast client proposed in the third embodiment of the present application. Detailed implementation manners

[0045] In order to make the objectives, technical solutions and advantages of the present application more clear and understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts shall fall within the protection scope of the present application.

[0046] It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between various embodiments may be combined with each other, but it must be based on the fact that those of ordinary skill in the art can implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the protection scope required by the present application.

[0047] Please refer to Figure 1 , Figure 1 An application environment architecture diagram for implementing various embodiments of the present application. The present application can be applied to an application environment including, but not limited to, a test end 2, a client end 4, and a server end 6.

[0048] Among them, the test terminal 2 is used to play a test video and superimpose and display a clock in the playing screen. The clock is a millisecond-level clock. Preferably, the test terminal 2 can simultaneously superimpose and display two millisecond-level clocks in the playing screen. For example, the two millisecond-level clocks are respectively displayed above and below the left edge of the playing screen.

[0049] The test terminal 2 can be any electronic device capable of playing videos, such as a PC (Personal Computer), mobile phone, tablet computer, portable computer, etc.

[0050] The client 4 is used to use the live broadcast APP to capture the playing screen of the test terminal 2 playing the test video and forward the captured video stream to the server 6. The client 2 can be a mobile terminal device such as a mobile phone or a tablet computer.

[0051] The server 6 can be a local server established based on transmission protocols available for live broadcast such as RTMP (Real Time Messaging Protocol), HTTP (Hyper Text Transfer Protocol)-FLV (Flash Video), HLS (HTTP Live Streaming), WebRTC (Web Real-Time Communications). The server 6 is used to obtain the video stream and save video recordings. For example, a video recording is saved every 15 seconds. And multiple processes are started to analyze the saved video recordings simultaneously, output the detected time series, determine the lag of the video stream according to the time series, and upload and save the detection results. The server 6 can be a computing device such as a rack-mounted server, blade server, tower server, or cabinet server, and can be an independent server or a server cluster composed of multiple servers.

[0052] The purpose of the embodiment of the present application is to detect the lag in live broadcast streaming caused by the live broadcast APP of the client 4 to test whether the live broadcast APP meets the requirements in terms of lag metrics. It should be noted that in order to exclude the factor of live broadcast lag caused by unstable streaming services, the client 4 and the server 6 are only connected through local area network communication for data transmission and interaction. Refer to Figure 2 As shown, it is another form of application environment architecture diagram for implementing various embodiments of the present application. From Figure 2 it can be seen that the video stream captured by the client 4 is only pushed to the local server 6 through the local area network and does not pass through an external streaming media server for data transmission.

[0053] Currently, the detection schemes for live broadcast lags mainly include manual detection, third-party detection SDKs, data logging, data stream monitoring, etc. Among them, manual detection requires testers to experience and test for a long time to give the most appropriate lag report for the user experience. This lag detection method is closer to the actual experience, but it will consume a large amount of manpower, and there will be misjudgments by the human eye.

[0054] Third-party detection collects application data through an SDK, then uploads the data to a third-party server for statistical analysis, and then locates the cause of the lag. Data logging is to monitor the performance indicators of the live broadcast APP and report them through an interface. However, since an SDK package needs to be integrated into the live broadcast APP, the third-party detection method will increase the size of the live broadcast APP and cannot guarantee data security. Moreover, both data logging and third-party detection run in the live broadcast APP, and the accuracy of the detection results is poor.

[0055] Data stream monitoring detects live broadcast anomalies by monitoring the abnormal state of the live broadcast data stream, and cannot rule out lags caused by poor network stability of the online streaming media server.

[0056] Therefore, in order to conveniently and accurately detect the live broadcast lags caused by the client live broadcast APP in the embodiments of the present application, the application environment is controlled in a local area network, and the lag causes caused by the network stability of the online streaming media server can be excluded. Moreover, by displaying a clock in the playback screen of the test terminal 2 and then having the client 4 capture the playback screen and perform local streaming, the server 6 can accurately detect the lag through the analysis of the time series in the acquired video stream, and can determine that the lag is caused by the client 4 performing the streaming.

[0057] Embodiment 1

[0058] As Figure 3 shown, it is a flowchart of a method for detecting lags in a live broadcast client proposed in the first embodiment of the present application. It can be understood that the flowchart in the embodiments of this method is not used to limit the order of execution steps. If necessary, some steps in this flowchart can also be added or deleted. The following uses the server as the execution subject to illustrate this method.

[0059] This method includes the following steps:

[0060] S200, obtain the video stream pushed by the client through the local area network, where the video stream is the playback screen of the test video played by the test terminal captured by the client, and a pre-set clock is superimposed and displayed in the playback screen.

[0061] When it is necessary to conduct a stuttering test on the live broadcast APP of the client, a connection is established through the local area network between the test end, the client, and the server to create an application environment for pushing and pulling streams under the local area network. When the test starts, first, a pre-prepared test video is played on the display screen of the test end, and a pre-set clock is simultaneously displayed in the playback screen. The clock is a millisecond-level clock and is displayed at any predetermined position on the display screen. The time information on the clock can provide a basis for subsequent judgment of whether the live video stream is stuttering.

[0062] Then, the live broadcast APP in the client is used to capture the playback screen of the test end, and the captured video stream is pushed to the server. Specifically, the client sends a stream pushing request in the local area network. After receiving the stream pushing request, the server returns a stream pushing address to the client, and then the client pushes the video stream to the server according to the stream pushing address.

[0063] However, when the camera of the client captures the changing clock on the display screen of the test end, there is a certain probability that it will be exposed when the display screen refreshes the clock, resulting in Figure 4 ghosting and afterimage pictures as shown, which will greatly increase the difficulty for the subsequent server to identify the time information in the video stream. If the identification is incorrect, it is easy to lead to misjudgment of stuttering.

[0064] The mechanism of the display screen refresh is that each frame of the image is continuously scanned line by line in sequence by the electron beam, that is, progressive scanning. This process is very fast and cannot be perceived by the human eye, but the process of the display screen progressive refresh is often captured in video shooting, that is, part of the screen has been refreshed while the other part has not. Therefore, to address the problem of the failure of time information recognition caused by the afterimage and ghosting due to the display screen refresh, in a preferred embodiment, multiple clocks can be set at multiple positions on the display screen of the test end. When one clock is in the position of being refreshed and an afterimage appears, another clock can remain clear because it has not been refreshed yet. After testing, two millisecond-level clocks can be used and are respectively set above and below the left edge of the display screen. And, the times displayed by the two clocks are kept consistent.

[0065] S202, Extract the time series from the video stream.

[0066] In this embodiment, the time series refers to the time information on the clock displayed in the playback screen. Specifically, the time information on the clock can be extracted from the video stream through OCR (Optical Character Recognition) technology. OCR refers to the process in which an electronic device examines the characters printed on paper and then translates the recognized shapes into computer text using character recognition methods; that is, the process of scanning text materials and then analyzing and processing the obtained image files to obtain text and layout information. In this embodiment, character recognition is performed on each frame of video image in the video stream, especially on the area where the clock is located, and the recognized shapes are translated into computer text, that is, time information.

[0067] After pulling the stream, the server saves a video recording every predetermined time interval, for example, saves a video recording every 15 seconds. Then, OCR technology is used to recognize the time information on the clock in each frame of the video recording, and the time information recognized from each frame of the video image is sorted in the playback order to obtain a time series.

[0068] Preferably, for the case where two millisecond-level clocks are set in the display screen, the server cuts each frame of the video image into two clock areas for OCR recognition respectively. And, regular matching and confidence filtering are performed on each recognition result, and the recognition result with the highest confidence is selected as the time information of this frame of the video image. The regular matching refers to judging whether the recognized time information conforms to the clock format. For example, the numbers corresponding to hours can only be 0-23, and the numbers corresponding to minutes and seconds can only be 0-59. If the recognized time information does not conform to the clock format, it means that the recognition result is incorrect. In addition, when using OCR technology to recognize the time information, in addition to outputting the recognition result, the corresponding confidence level, that is, the credibility of the recognition result, will also be output. The confidence filtering means that when the output confidence level is less than a preset threshold, for example, less than 90%, it is judged that the recognition result is not credible, that is, the recognition result is incorrect. When two clocks are set, two recognition results can be obtained through OCR recognition in each frame of the video image. When neither of the two recognition results is incorrect, one of the recognition results with a higher confidence is selected as the time information extracted from this frame of the video image and returned, that is, the valid time series is preferentially extracted.

[0069] In addition, if the difference between the two recognition results of the same frame of the video image is too large, for example, greater than 100 milliseconds, it can also be determined that the OCR recognition result is incorrect. Because in the same type of video image, the time information difference between the two clocks caused only by screen refresh is very small.

[0070] In this embodiment, frames with OCR recognition errors can be defaulted to normal frames, because it is normal for a live video stream to not have freezes, while freezes are abnormal. In addition, since the live video stream is orderly, it is also necessary to correct the outliers in the identified time series before outputting the final result.

[0071] S204: Perform freeze determination on the video stream according to the time sequence.

[0072] Since the time information on the clock in the playback screen of the test end increases in sequence as the test video is played, under normal circumstances, the time series extracted by the server should also increase in sequence, otherwise it means that the video stream may be stuck.

[0073] In this embodiment, when the time difference between frames is greater than a preset value, for example, when the time information difference between the current frame of video image and the previous frame of video image is greater than 200 milliseconds, it is determined that the video stream is stuck at the current frame of video image.

[0074] It is worth noting that due to the large amount of video stream data, for example, when there are 30 frames of video images per second, a 15-minute video stream will have 27,000 images to be analyzed, resulting in the OCR recognition speed and the jamming analysis speed not being able to meet the system requirements, which will cause the video to be detected to pile up. Therefore, in a preferred embodiment, multiple processes can be used for streaming, recognition and analysis. For example, one process runs streaming, three processes run OCR recognition and jamming judgment, and JoinableQueue is used between processes to communicate. The processing method of starting multiple processes at the same time can greatly improve the detection speed of the server.

[0075] Preferably, during the entire detection process, the server also needs to monitor the network status of the local area network (mainly between the server and the client) to eliminate freezes caused by unstable network status of the local area network, and ensure that the source of the problem caused by the live broadcast APP of the client can be effectively located when freezes occur. The specific monitoring method can be: after the server obtains the IP (Internet Protocol) address of the client, it continuously sends ping requests to the client according to the IP address, for example, 20 requests per second, and records the return time. If the return time is not timed out, it means that the network status between the client and the server is normal.

[0076] S206, saving the determination result and displaying it on the front end.

[0077] In this embodiment, the test video, the video recording, the time series, and the results of stutter determination can all be retained for subsequent review and problem troubleshooting. Among them, they can be uploaded to other servers for storage, such as a database server, or they can be stored directly on the server side. The review can be manual review, which is carried out according to the time information of the clock in the video image of the stutter frame and the video image of the previous frame to determine whether the stutter frame automatically determined actually has stuttering.

[0078] The server side can also provide multiple interfaces to facilitate the front end to display the stutter determination results in multiple dimensions. The front end can also provide an interface for users to start, end, edit, or delete detection tasks, as well as view the task progress and detection results, etc. The editing includes modifying parameters such as the model information and the test time.

[0079] In addition, since the essence of the ghosting and afterimage problems is that the display screen of the test end is captured by the client during the refresh process, the speed at which the display screen completely refreshes a frame is closely related to the probability of afterimage generation. And GTG (GreyTo Grey, gray-scale response time) is an index that describes the refresh speed of the display screen. The response time refers to the reaction speed of the display screen to the input signal, that is, the time for the liquid crystal particles to change from dark to bright or from bright to dark. The GTG represents the time required for a pixel to change between two colors, and the faster this response speed is, the more it can reduce the motion blur of the screen.

[0080] Theoretically speaking, the smaller the GTG value of the display screen is, the more significantly the occurrence of ghosting and afterimages can be reduced. To further improve the accuracy of stutter detection, through a comparative exploration experiment on the display problem of the clock, it is found that updating the display device of the test end can significantly reduce the misjudgment of the server side. As Figure 5 shown, it is a schematic diagram of the results of a comparative exploration experiment on a clock display method. Among them, the detected stutter count is the average number of stutters detected by the server side per hour, and the number of misjudgments is the average number of misjudgments per hour obtained by verifying the stutter determination results through manual review. The experiment is divided into three groups, and different display models and clock display fonts are tested respectively. According to the experimental results, it can be seen that using a display with a smaller GTG and a more recognizable font can significantly improve the accuracy of time series recognition and reduce misjudgments. In this embodiment, the display screen of the test end 2 can use a device with a GTG index of 2 milliseconds. If it is possible to use a device with a GTG index of 0.5 milliseconds, it is expected that there will still be some room for improvement in the detection accuracy.

[0081] The live client freeze detection method proposed in this embodiment can form a local area network, and display a clock superimposed on the screen of the test video played on the test end. After the client captures the playback screen of the test end through the live broadcast APP, it pushes the stream to the server through the local area network. The server uses OCR technology to extract the time series in the live video stream, so as to automatically detect the live freeze situation caused by the client. Moreover, the effective time series can be extracted by selecting the better one from two clocks, greatly reducing the misjudgment rate of recognition. The video recording and judgment results of this embodiment can be retained for convenient review and problem troubleshooting.

[0082] Compared with manual detection, this embodiment uses automatic detection to analyze each frame of the video stream, which can save human resources and the conclusion is more reliable and objective. Compared with data embedding and third-party detection SDK, this embodiment has a more comprehensive and accurate coverage, and is closer to the effect presented to users. Compared with data stream monitoring, this embodiment is dedicated to monitoring the push stream freeze caused by the live broadcast APP of the client, which can exclude the interference of the online streaming media server on the detection result and facilitate problem location.

[0083] After detecting the freeze problem caused by the live broadcast APP of the client, the live broadcast APP can be further improved to optimize the user experience and promote the retention of users (mainly mobile client anchors).

[0084] Embodiment 2

[0085] As Figure 6 shown, it is a schematic diagram of the hardware architecture of an electronic device 20 proposed in the second embodiment of the present application. In this embodiment, the electronic device 20 may include, but is not limited to, a memory 21, a processor 22, and a network interface 23 that can communicate with each other through a system bus. It should be noted that Figure 6 only the electronic device 20 with components 21-23 is shown, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. In this embodiment, the electronic device 20 may be the server.

[0086] The memory 21 at least includes one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disc, etc. In some embodiments, the memory 21 may be an internal storage unit of the electronic device 20, such as the hard disk or memory of the electronic device 20. In other embodiments, the memory 21 may also be an external storage device of the electronic device 20, such as a plug-in hard disk equipped on the electronic device 20, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Of course, the memory 21 may also include both the internal storage unit of the electronic device 20 and its external storage device. In this embodiment, the memory 21 is generally used to store the operating system installed on the electronic device 20 and various application software, such as the program code of the live client lag detection system 60, etc. In addition, the memory 21 may also be used to temporarily store various data that have been output or will be output.

[0087] In some embodiments, the processor 22 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 22 is generally used to control the overall operation of the electronic device 20. In this embodiment, the processor 22 is used to run the program code stored in the memory 21 or process data, such as running the live client lag detection system 60, etc.

[0088] The network interface 23 may include a wireless network interface or a wired network interface, and the network interface 23 is generally used to establish a communication connection between the electronic device 20 and other electronic devices.

[0089] Embodiment III

[0090] As Figure 7 shown, a module schematic diagram of a live client lag detection system 60 is proposed in the third embodiment of this application. The live client lag detection system 60 can be divided into one or more program modules. One or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiments of this application. The program modules referred to in the embodiments of this application refer to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment.

[0091] In this embodiment, the live client freezing detection system 60 includes:

[0092] An acquisition module 600, configured to acquire a video stream pushed by the client through a local area network, where the video stream is a playback picture of a test video played by a test terminal captured by the client, and a pre-set clock is superimposed and displayed in the playback picture.

[0093] When it is necessary to perform a freezing test on the live APP of the client, connect the test terminal, the client, and the server through a local area network to establish an application environment for pushing and pulling streams under the local area network. When the test starts, first play a pre-prepared test video on the display screen of the test terminal, and simultaneously display a pre-set clock in the playback picture. The clock is a millisecond-level clock and is displayed at any predetermined position on the display screen. The time information on the clock can provide a basis for subsequent judgment of whether the live video stream is frozen.

[0094] Then, use the live APP in the client to capture the playback picture of the test terminal, and push the captured video stream to the server. Specifically, the client sends a stream pushing request in the local area network. After receiving the stream pushing request, the server returns a stream pushing address to the client, and then the client pushes the video stream to the server according to the stream pushing address.

[0095] To address the problem of invalid time information recognition caused by afterimages and ghosting due to display screen refreshing, in a preferred embodiment, multiple clocks can be set at multiple positions on the display screen of the test terminal. When an afterimage appears at the position where one clock is located due to refreshing, another clock can remain clear because it has not been refreshed yet. After testing, two millisecond-level clocks can be used, which are respectively set above and below the left edge of the display screen. And the times displayed by the two clocks are kept consistent.

[0096] An extraction module 602, configured to extract a time series from the video stream.

[0097] In this embodiment, the time series refers to the time information on the clock displayed in the playback picture. Specifically, the time information on the clock can be extracted from the video stream through OCR technology. Perform character recognition on each frame of video image in the video stream, especially perform character recognition on the area where the clock is located, and translate the recognized shape into computer text, that is, time information.

[0098] After pulling the stream, the obtaining module 600 saves a video recording every predetermined time, for example, saves a video recording every 15 seconds. Then, the extraction module 602 uses OCR technology to identify the time information on the clock in each frame of the video recording, and sorts the time information identified from each frame of the video image in the playback order to obtain a time series.

[0099] Preferably, for the case where two millisecond-level clocks are set in the display screen, the extraction module 602 cuts each frame of the video image into two clock regions for OCR recognition respectively. And, perform regular matching and confidence filtering on each recognition result, and select the recognition result with the highest confidence as the time information of this frame of the video image. The regular matching refers to judging whether the recognized time information conforms to the clock format. For example, the numbers corresponding to hours can only be 0-23, and the numbers corresponding to minutes and seconds can only be 0-59. If the recognized time information does not conform to the clock format, it means that the recognition result is incorrect. In addition, when using OCR technology to recognize the time information, in addition to outputting the recognition result, the corresponding confidence level will also be output, that is, the credibility of the recognition result. The confidence filtering means that when the output confidence level is less than a preset threshold, for example, less than 90%, it is judged that the recognition result is not credible, that is, the recognition result is incorrect. When two clocks are set, two recognition results can be obtained through OCR recognition in each frame of the video image. When neither of the two recognition results is in error, select one of the recognition results with a higher confidence as the time information extracted from this frame of the video image and return it, that is, preferentially extract the effective time series.

[0100] In addition, if the two recognition results of the same frame of the video image differ too much, for example, more than 100 milliseconds, it can also be determined that the OCR recognition result is incorrect. Because in the same kind of video image, the time information difference between the two clocks caused only by screen refresh is very small.

[0101] For the frames with incorrect OCR recognition, in this embodiment, they can be defaulted to normal frames, because for a live video stream, it is normal not to have stuttering, while stuttering is abnormal. And, since the live video stream is ordered, it is also necessary to correct the outliers in the recognized time series and then output the final result.

[0102] The determination module 604 is used to perform stuttering determination on the video stream according to the time series.

[0103] Since in the playback screen at the test end, the time information on the clock increases in sequence as the test video is played. Therefore, under normal circumstances, the time series extracted by the extraction module 602 should also increase in sequence, otherwise it means that the video stream may be stuttering.

[0104] In this embodiment, when the inter-frame time difference is greater than a preset value, for example, when the time information difference between the current frame video image and the previous frame video image is greater than 200 milliseconds, the determination module 604 determines that the video stream has a freeze at the current frame video image.

[0105] It should be noted that due to the excessive amount of video stream data, for example, when there are 30 frame video images per second, a 15-minute video stream will have 27,000 pictures to be analyzed, resulting in the OCR recognition speed and freeze analysis speed not meeting the system requirements, causing a backlog of videos to be detected. Therefore, in a preferred embodiment, multiple processes can be used for pulling the stream, recognition, and analysis. For example, one process runs for pulling the stream, and three processes run for OCR recognition and freeze determination, and JoinableQueue is used for communication between processes. The processing method of starting multiple processes simultaneously can greatly improve the detection speed of the server.

[0106] The saving module 606 is used to save the determination result and display it on the front end.

[0107] In this embodiment, the test video, the video recording, and the results of freeze determination can all be retained for subsequent review and problem troubleshooting. Among them, it can be uploaded to other servers for storage, such as a database server, or it can be stored directly on the server. The review can be manual review, and based on the time information of the clock in the video image of the freeze frame and the previous frame video image, it is determined whether the freeze frame automatically determined actually has a freeze.

[0108] The server can also provide multiple interfaces to facilitate the front end to display the freeze determination results from multiple dimensions. The front end can also provide an interface for users to start, end, edit, or delete detection tasks, as well as view the task progress and detection results, etc. The editing includes modifying parameters such as model information and test time.

[0109] The live client freeze detection system proposed in this embodiment can form a local area network, and in the test end, a clock is superimposed and displayed on the playing screen of the test video. After the client captures the playing screen of the test end through the live broadcast APP, it pushes the stream to the server through the local area network. The server uses OCR technology to extract the time sequence in the live video stream to automatically detect the live freeze situation caused by the client. Moreover, the effective time sequence can be extracted by selecting the better of the two clocks, greatly reducing the recognition misjudgment rate. The video recording and determination results of this embodiment can be retained to facilitate review and problem troubleshooting.

[0110] Embodiment 4

[0111] The present application also provides another implementation manner, that is, to provide a computer-readable storage medium storing a live client llag detection program, which can be executed by at least one processor, so that the at least one processor executes the steps of the live client llag detection method as described above.

[0112] It should be noted that in this text, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such process, method, article or device. Without more limitations, an element de ned by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the element.

[0113] The serial numbers of the embodiments of the present application above are only for description and do not represent the superiority or inferiority of the embodiments.

[0114] Obviously, those skilled in the art should understand that the above-mentioned modules or steps of the embodiments of the present application can be implemented by a general-purpose computing device. They can be concentrated on a single computing device or distributed on a network composed of multiple computing devices. Optionally, they can be implemented by program codes executable by the computing device, so that they can be stored in a storage device and executed by the computing device. And in some cases, the steps shown or described can be executed in a di erent order than here, or they can be separately fabricated into individual integrated circuit modules, or multiple modules or steps among them can be fabricated into a single integrated circuit module to implement. Thus, the embodiments of the present application are not limited to any speci c combination of hardware and software.

[0115] The above are only the preferred embodiments of the embodiments of the present application, and do not limit the patent scope of the embodiments of the present application. Any equivalent structure or equivalent process transformation made by using the speci cations and drawings of the embodiments of the present application, or directly or indirectly applied in other related technical elds, are equally included in the patent protection scope of the embodiments of the present application.

Claims

1. A method for detecting lags in a live broadcast client, which is applied to a server, characterized in that, The server is communicatively connected to the client and the test end through a local area network. The method includes: Obtaining a video stream pushed by the client through the local area network. The video stream is a playback picture of the test video played by the test end captured by the client, and a pre-set clock is superimposed and displayed in the playback picture. Extracting a time series from the video stream. The time series is the time information on the clock. Wherein, one time series is obtained by sorting the time information on the clock of each video image in the video stream in the playback order. Judging whether the video stream has lags according to whether the time series increases in order.

2. The live client freeze detection method according to claim 1, wherein The clock is a millisecond-level clock displayed at any predetermined position on the display screen of the test end.

3. The live client freeze detection method according to claim 2, wherein The clock includes multiple clocks displayed at multiple predetermined positions on the display screen of the test end, and the time displayed by the multiple clocks is consistent.

4. The live client freezing detection method according to claim 3, wherein, The clock includes two millisecond-level clocks displayed above and below the left edge of the display screen of the test end.

5. The live client freezing detection method according to claim 1, characterized in that, The obtaining of the video stream pushed by the client through the local area network includes: Receiving a stream pushing request sent by the client in the local area network and returning a stream pushing address to the client. Obtaining the video stream pushed by the client according to the stream pushing address.

6. The live client freezing detection method according to claim 1, wherein The extracting of the time series from the video stream includes: Using optical character recognition technology to recognize the time information of the clock from each video image of the video stream to obtain the time series.

7. The live client freeze detection method according to claim 1, wherein The extracting of the time series from the video stream includes: Saving a video recording of the video stream every predetermined time. Using optical character recognition technology to recognize the time information on the clock from each video image of the video recording. Sorting the recognized time information in the playback order of the video recording to obtain the time series.

8. The live client freeze detection method according to claim 4, characterized in that The extracting of the time series from the video stream includes: Cutting each video image into two clock regions. Respectively using optical character recognition technology to recognize the time information on the clock for the two clock regions to obtain two recognition results. Selecting the one with the highest confidence from the two recognition results as the time information recognized for this frame of video image.

9. The live client freeze detection method according to claim 1, wherein The judging whether the video stream has lags according to whether the time series increases in order includes: In the case where the time information difference between this frame of video image and the previous frame of video image is greater than a preset value, it is judged that the video stream has lags.

10. The live client freezing detection method according to claim 1, wherein The method further includes: Saving the judgment result, the test video, the video recording, and the time series for review, and displaying the judgment result at the front end.

11. The live client freeze detection method according to claim 1, characterized in that The method further includes: Simultaneously using multiple processes to separately run the processes of obtaining the video stream, extracting the time series, and judging lags.

12. A live broadcast client lag detection system, characterized in that, The system includes: An obtaining module, configured to obtain a video stream pushed by a client through a local area network. The video stream is a playback picture of a test video played by the test end captured by the client, and a pre-set clock is superimposed and displayed in the playback picture. An extraction module, configured to extract a time series from the video stream, where the time series is time information on the clock; wherein, one time series is obtained by sorting the time information on the clock of each video image in the video stream in the playing order. A determination module, configured to determine whether the video stream is stuck according to whether the time series increases in order.

13. An electronic device, characterized in that, The electronic device includes: a memory, a processor, and a live client stuck detection program stored on the memory and executable on the processor. When the live client stuck detection program is executed by the processor, the live client stuck detection method according to any one of claims 1 to 11 is implemented.

14. A computer-readable storage medium, characterized in that, A live client stuck detection program is stored on the computer-readable storage medium. When the live client stuck detection program is executed by the processor, the live client stuck detection method according to any one of claims 1 to 11 is implemented.

15. A computer program product, the computer program product comprising computer instructions, characterized in that, When the computer instruction is executed by the processor, the live client stuck detection method according to any one of claims 1 to 11 is implemented.

Images