Virtual terminal-based visual monitoring method and device, and electronic equipment

Abstract

The application discloses a virtual terminal-based visual monitoring method and device and electronic equipment, and belongs to the technical field of personal home IPTV. The method comprises the following steps: receiving and playing a target video corresponding to a video stream of a channel to be monitored by using a virtual terminal; identifying whether a monitoring index of the played target video is abnormal; acquiring an alarm notification corresponding to the target video in the case that it is identified that the monitoring index of the played target video is abnormal; and displaying the target video and the alarm notification. In the above manner, the cost can be reduced.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of personal home IPTV, and particularly relate to a visual monitoring method and device based on a virtual terminal and an electronic device. BACKGROUND

[0002] With the development of Internet technology and the increasing demand for home video entertainment, the IPTV / OTT video platform can provide more multimedia service functions and resource updates for users due to its combination with the Internet and integration of related resources, and gradually becomes the mainstream of the market.

[0003] In the related art, the IPTV / OTT video platform is generally monitored by using a traditional manual monitoring mode. This scheme is based on a real terminal, and the hardware investment and maintenance cost of the monitoring device are high. SUMMARY

[0004] Embodiments of the present application provide a visual monitoring method and device based on a virtual terminal and an electronic device, which can solve the problem of high cost.

[0005] To solve the above technical problems, the present application is implemented as follows:

[0006] In a first aspect, the embodiments of the present application provide a visual monitoring method based on a virtual terminal, which includes: receiving and playing a target video corresponding to a video stream of a to-be-monitored channel by using a virtual terminal; identifying whether a monitoring indicator of the played target video is abnormal; in a case where it is identified that the monitoring indicator of the played target video is abnormal, acquiring an alarm notification corresponding to the target video; and displaying the target video and the alarm notification.

[0007] In a second aspect, the embodiments of the present application provide a visual monitoring device based on a virtual terminal, which includes: a playing module configured to receive and play a target video corresponding to a video stream of a to-be-monitored channel by using a virtual terminal; an identifying module configured to identify whether a monitoring indicator of the played target video is abnormal; an acquiring module configured to acquire an alarm notification corresponding to the target video in a case where it is identified that the monitoring indicator of the played target video is abnormal; and a displaying module configured to display the target video and the alarm notification.

[0008] In a fifth aspect, the embodiments of the present application provide an electronic device including a processor and a memory. The memory stores programs or instructions that can be run on the processor. When the programs or instructions are executed by the processor, the steps of the method according to the first aspect described above are implemented.

[0009] In a sixth aspect, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium storing a program or instructions, the program or instructions being executed by a processor to implement the steps of the method in the first aspect.

[0010] In a seventh aspect, an embodiment of the present application provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions that, when executed by a computer, cause the computer to perform the steps of the method in the first aspect.

[0011] The technical solutions provided by the present application can include the following beneficial effects:

[0012] In the embodiments of the present application, the virtual terminal can be used to receive and play a target video corresponding to a video stream of a to-be-monitored channel, identify whether a monitoring index of the played target video is abnormal, acquire an alarm notification corresponding to the target video in a case where it is identified that the monitoring index of the played target video is abnormal, and display the target video and the alarm notification. In this way, the cost can be reduced.

[0013] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0014] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present application and, together with the specification, serve to explain the principles of the present application.

[0015] Figure 1 Fig. 1 shows a flowchart of a visual monitoring method based on a virtual terminal according to an embodiment of the present application;

[0016] Figure 2a Fig. 2 shows a flowchart of an image recognition method according to an embodiment of the present application;

[0017] Figure 2b Fig. 3 shows a flowchart of another image recognition method according to an embodiment of the present application;

[0018] Figure 3 Fig. 4 shows a schematic diagram of a visual monitoring system based on a virtual terminal according to an embodiment of the present application;

[0019] Figure 4 Fig. 5 shows a schematic diagram of another visual monitoring method based on a virtual terminal according to an embodiment of the present application;

[0020] Figure 5 Fig. 6 shows a schematic diagram of a video stream merging according to an embodiment of the present application.

[0021] Figure 6 A structure schematic diagram of a visual monitoring device based on a virtual terminal is shown;

[0022] Figure 7 A structure schematic diagram of an electronic device is shown;

[0023] Figure 8 A structure schematic diagram of another electronic device is shown. DETAILED DESCRIPTION

[0024] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The following description is made with reference to the accompanying drawings in which like reference numerals refer to like elements in the several figures. The following description of exemplary embodiments is not representative of all possible embodiments consistent with the present application. Rather, it is merely intended to describe some embodiments consistent with the present application in sufficient detail to enable one skilled in the art to practice the application as detailed in the appended claims.

[0025] With the development of Internet technology and the increasing demand for home video entertainment, traditional cable television gradually withdraws from the current home video entertainment market. IPTV / OTT (over the top, which refers to providing various application services to users through the Internet) video platform gradually becomes the mainstream of the market because of its combination with the Internet and the integration of related resources, which can provide more multimedia service functions and resource updates for users.

[0026] IPTV is an interactive network television, which is a new media technology integrating Internet, multimedia and communication technology, using broadband network as a medium to provide a variety of interactive services including digital television for home users, with a series of functions such as live broadcast, on-demand, time shift, and watching, to realize the substantial interaction between media providers and media consumers.

[0027] OTT is an Internet television, which is an Internet television content service platform and an integrated service platform, and can provide a variety of application services to users through the Internet.

[0028] In the related art, on the one hand, video monitoring is still based on real terminals; on the other hand, although IPTV / OTT video platform has greater technical breakthroughs compared to cable television, the operation and maintenance of IPTV / OTT video platform also has the following defects:

[0029] I. High cost of hardware investment and maintenance of monitoring equipment such as set-top box hardware, content delivery network (CDN) code stream monitoring equipment, etc.

[0030] II. IPTV / OTT video resource compliance requirements are high, and it is difficult to have universality. Monitoring and quality difference judgment are complex, deployment is cumbersome, monitoring is complex, upgrade cycle is long, and manual operation workload is greatly increased. In actual application, with the rapid growth of user scale, the corresponding user data also presents explosive growth, so the multiple peripheral systems and business chains involved make the manual operation workload greatly increased; since the IPTV / OTT video platform contains integration of diversified video resources, therefore, in order to guarantee the auditing process of resource compliance, there are great challenges, and it is complex to monitor and judge the quality difference of hundreds of live channels. In related technologies, the method of combining display of multiple set-top boxes large screens exists the problems of cumbersome deployment and complex monitoring. Since the version of the IPTV / OTT video platform needs to be tested and verified respectively for all network CDN nodes, there is a defect of long upgrade cycle. Since the IPTV / OTT video platform realizes data interaction based on the Internet and the end-to-end video playback form, therefore, combined with the special properties of large screen content, multiple means are needed to guarantee content security, so that the requirement of video content compliance is too high, and it is difficult to have universality.

[0031] Figure 1 A flowchart of a visual monitoring method based on a virtual terminal provided by an example embodiment of the present application is shown. The method 100 can be executed by an electronic device, which can be a terminal device such as a computer. As shown in the figure, the method mainly includes the following steps: Figure 1

[0032] S101: receiving and playing a target video corresponding to a video stream of a to-be-monitored channel by using a virtual terminal.

[0033] In the embodiment of the present application, the virtual terminal can be used to receive and play the target video corresponding to the video stream of the to-be-monitored channel. The to-be-monitored channel can be determined by simulating the playback behavior of a set-top box user. In actual application, the to-be-monitored channel can be predicted according to a trained model, or can be reproduced according to the actual playback behavior of a user, and the embodiment of the present application does not make specific limitation. The purpose of simulating user playback is to better cover more possibilities. The virtual terminal can be a virtual set-top box, which can receive and play the target video corresponding to the video stream of the to-be-monitored channel through the virtual set-top box. In this way, there is no need for a physical set-top box, which can save costs.

[0034] In an optional implementation manner, the step of receiving and playing the target video corresponding to the video stream of the to-be-monitored channel by using the virtual terminal can include the following steps:

[0035] ​Step 1, obtaining monitoring task configuration information corresponding to the virtual terminal, wherein the monitoring task configuration information includes the play address information of the to-be-monitored channel; in the embodiment of the present application, the monitoring task configuration information corresponding to the virtual terminal obtained can include the play address information of the to-be-monitored channel, and can also include the monitoring task. The play address information of the to-be-monitored channel can include the play address of the to-be-monitored channel and the information of the CDN node to be monitored, and the CDN node can be specified by a person. In the embodiment of the present application, the quality of service of the CDN node in a specified area can be monitored in real time.

[0036] Step 2, accessing the content distribution network (CDN) node corresponding to the play address information by using the virtual terminal, receiving and playing the target video corresponding to the video stream of the to-be-monitored channel. In the embodiment of the present application, the virtual terminal can be used to access the corresponding content distribution network (CDN) node to obtain the video stream of the to-be-monitored channel, and then the target video corresponding to the video stream of the to-be-monitored channel is played in the background.

[0037] S102: identifying whether the monitoring indicators of the played target video are abnormal.

[0038] In the embodiment of the present application, whether the monitoring indicators of the played target video are abnormal can be identified, so as to realize live quality and content security monitoring, and quickly locate live lag, flow interruption, audio and video asynchronization, live content tampering and other faults. In actual application, the monitoring indicators can be determined according to the quality of experience (QoE) of the picture and the key performance indicator (KPI) quality index of the Internet protocol (Internet Protocol, IP) layer.

[0039] In an optional implementation, the monitoring indicators include the play quality; the identification of whether the monitoring indicators of the played target video are abnormal includes:

[0040] collecting the play log of the target video through the virtual terminal;

[0041] identifying whether the play quality of the played target video is abnormal by analyzing the play log.

[0042] In the embodiments of the present application, the playing log of the target video can be collected through the client Android application package (APK) of the virtual set-top box; then, by analyzing the playing log, it is identified whether the playing quality of the target video is abnormal, that is, whether there is live quality difference, such as mosaic, freezing, black screen, etc. In actual application, visual artificial intelligence (AI) model can be used to identify playing failure. The visual AI model has the ability of image recognition of computational vision (CV), by constructing sample pictures of common live failure & playing quality difference, collecting training corpus of image recognition model, and learning defect picture annotation. At the same time, the CV small model capability library is released periodically, and the recognition range and accuracy are improved through continuous iteration and upgrading of the visual AI model. Among them, the defect pictures used for model training include but are not limited to: large-area black screen, default filling poster, and screen area. Without manual recognition, not only the efficiency and cost can be improved, but also the accuracy can be improved.

[0043] In an optional implementation, the monitoring index includes playing content; and the identifying whether the monitoring index of the target video being played is abnormal includes:

[0044] At least one frame of video image of the target video is acquired;

[0045] By identifying the content of at least one frame of the video image, it is identified whether the playing content of the target video is abnormal.

[0046] In the embodiments of the present application, at least one frame of video image of the target video can be acquired according to the monitoring task; then, by identifying the content of at least one frame of video image, it is identified whether the playing content of the target video is abnormal, such as whether there is error recognition, station logo recognition, text recognition, etc., and the picture quality can also be analyzed.

[0047] In an optional implementation, the identifying whether the playing content of the target video is abnormal by identifying the content of at least one frame of the video image can include the following steps: Figure 2a

[0048] ​S201: determining the video image as a first abnormal video frame or a first normal video frame by classifying the video image by using the trained classification model; in the embodiment of the present application, at least one video image can be obtained, and then the trained classification model is used for preliminary classification, and then it is determined that the video image is a first abnormal video frame or a first normal video frame. The classification model can quickly identify the relatively obvious abnormal scene, such as a relatively large area of the picture showing an abnormal pop-up box, an abnormal error code, an abnormal loading of a poster, and the like. By identifying the video image by using the classification model, in the case that the video image has a relatively obvious abnormal scene, it can be determined that the video image is a first abnormal video frame; in the case that the video image does not have a relatively obvious abnormal scene, it can be determined that the video image is a first normal video frame.

[0049] S202: determining the first normal video frame as a second abnormal video frame or a second normal video frame by comparing the similarity of the first normal video frame with a reference picture; in the embodiment of the present application, if the video image has a small area error such as text description and language garbled code, it may be classified as a normal scene by the classification model, that is, it is determined that the video image is a first normal video frame, and then secondary screening is needed, that is, the similarity comparison with the reference picture is needed to determine that the first normal video frame is a second abnormal video frame or a second normal video frame. In the case that the similarity comparison result of the first normal video frame with the reference picture is greater than a set threshold, it can be determined that the first normal video frame is a second normal video frame, which indicates a normal scene; in the case that the similarity comparison result of the first normal video frame with the reference picture is less than the set threshold, it can be determined that the first normal video frame is a second abnormal video frame, which indicates an abnormal scene. By performing secondary analysis on the first normal video frame, the similarity ratio result can improve the identification accuracy of the small difference area, and further improve the accuracy of data analysis.

[0050] S203: determining the second abnormal video frame as a third abnormal video frame or a third normal video frame by calculating the difference value of the second abnormal video frame and identifying whether the difference area of the second abnormal video frame is a solid color; in the embodiment of the present application, for the abnormal scene picture (second abnormal video frame) screened out by S201 and S202, the difference value is calculated by using the opencv (an open source computer vision and machine learning software library) related interface, and the difference area is marked to determine whether the difference area is a solid color. If it is a solid color, it is classified as a normal scene, and it is determined that the second abnormal video frame is a third normal video frame; if it is not a solid color, it is classified as an abnormal scene, and it is determined that the second abnormal video frame is a third abnormal video frame.

[0051] S204: determining whether the third abnormal video frame is a fourth abnormal video frame or a fourth normal video frame by identifying whether the third abnormal video frame contains abnormal characters. In an embodiment of the present application, whether the third abnormal video frame contains abnormal characters can be identified. In actual application, the difference area in the third abnormal video frame which is not pure color can be used to extract the text therein by using optical character recognition (OCR), and it is determined whether the text contains abnormal characters. If the text does not contain abnormal characters, the third abnormal video frame can be classified into a normal scene, that is, the third abnormal video frame is determined to be a fourth normal video frame. If the text contains abnormal characters, the third abnormal video frame can be classified into an abnormal scene, that is, the third abnormal video frame is determined to be a fourth abnormal video frame.

[0052] S205: determining whether the playing content of the target video is abnormal according to the first abnormal video frame and the fourth abnormal video frame. In an embodiment of the present application, the first abnormal video frame and the fourth abnormal video frame are video images containing abnormalities, which can be used to indicate that the playing content of the target video contains abnormalities, and can further indicate what kind of abnormalities the playing content of the target video contains.

[0053] In an embodiment of the present application, the above identification process can also be as shown in FIG. 2B. Figure 2b In the process of identifying the video image, the mode of “coarse identification + fine analysis” can not only guarantee the analysis accuracy, but also reduce the occupation of computing resources. In S201, the abnormal images with obvious differences are divided by coarse identification, and then fine identification (further analysis of the first normal video frame) is performed based on the result of coarse identification. Compared with the related art in which all images are input into the difference analysis model, the embodiment of the present application can effectively reduce the pressure of model operation, and improve the rate of data analysis.

[0054] In another optional implementation, the video image can be distinguished according to whether there is an existing abnormal scene image in the process of image recognition. There are mainly two scenes, one with a reference image and one without a reference image. When the video image is a reference image, the similarity between the video image and the reference image can be compared directly to obtain a similarity, and a similarity threshold is used to determine whether it is a normal scene or an abnormal scene. In actual application, the similarity comparison algorithm provided by opencv can be used. When the video image is a reference image, the image classification is mainly performed through a model. At present, image classification has a relatively mature model, which is basically based on a convolutional neural network model, such as VGG16 (a powerful pre-trained model that can be used to identify the similarity between images), YOLOV3 (a real-time object detection algorithm that can identify specific objects in a video, real-time source or image), Single Shot MultiBox Detector (SSD), etc. In actual application, a VGG16 model can be used for moderate training to obtain an image classification model. The VGG16 model is trained through live fault samples for transfer learning, that is, the trained model is used for customized training with a small number of pictures. After transfer learning, a VGG16-based model is trained and saved as a file classify.keras. The model can be loaded directly for subsequent use. After loading the model, the model can be used to complete fault judgment on the obtained video image.

[0055] In the embodiments of the present application, the two image recognition methods described above are in a parallel relationship. In actual application, the two image recognition methods can be selected or used simultaneously, and the embodiments of the present application are not limited in this regard.

[0056] In an optional implementation, the picture detection technology based on deep learning can also be used to accurately and efficiently identify whether there are sensitive, advertising, logo (LOGOtype, Logo) watermark and other risk picture contents in the video image. The detection range of the picture detection technology based on deep learning is relatively larger than that of the step S204, and in actual application, the two methods can be used simultaneously, and the embodiments of the present application are not limited in this regard.

[0057] S103: In a case where it is identified that the monitoring indicator of the target video exists abnormity, acquiring an alarm notification corresponding to the target video.

[0058] In the embodiment of the present application, in the case that it is identified that the monitoring indicators of the target video being played are abnormal, the alarm notification corresponding to the target video can be obtained according to the preset alarm rule. The alarm rule can be that the quality indicator reaches the preset threshold. In actual application, the quality report corresponding to the monitoring task can also be generated. Not only can the real-time identification and analysis of the to-be-monitored channel be synchronized, improving the monitoring efficiency, but also the real-time requirement can be met, helping the video service quality guarantee and operation efficiency.

[0059] S104: display the target video and the alarm notification.

[0060] In the embodiment of the present application, the target video and the alarm notification can be displayed, thereby assisting the user in timely processing of the non-compliant content and guaranteeing the compliance of the video playing content.

[0061] In an optional implementation, the display of the target video and the alarm notification comprises:

[0062] The video streams played by the plurality of virtual terminals are merged to obtain a monitoring video;

[0063] The monitoring video is displayed on the monitoring screen, and the alarm notification is displayed on the monitoring screen.

[0064] In the embodiment of the present application, a plurality of to-be-monitored channels can be monitored at the same time, and then the target videos corresponding to the plurality of to-be-monitored channels are obtained. At this time, the video streams of the plurality of virtual terminals can be collected to be merged to obtain a monitoring video, and then the monitoring video is displayed on the monitoring screen, and the alarm notification corresponding to the target video is also displayed on the monitoring screen. In actual application, the alarm notification can be in the form of a pop-up window, which can be a modal pop-up window or a non-modal pop-up window, or other forms except the pop-up window, and the embodiment of the present application is not limited in this regard.

[0065] In the embodiment of the present application, the virtual terminal can be used to receive and play the target video corresponding to the video stream of the to-be-monitored channel; whether the monitoring indicators of the target video being played are abnormal is identified; in the case that it is identified that the monitoring indicators of the target video being played are abnormal, the alarm notification corresponding to the target video is obtained; and the target video and the alarm notification are displayed. Compared with related live monitoring means, the embodiment of the present application does not need to deploy a large number of physical terminals, greatly reducing the cost of hardware equipment investment and manual monitoring.

[0066] Figure 3A schematic diagram of a virtual terminal-based visual monitoring system is shown, which can include the following modules: live quality monitoring management module, cloud terminal management module, virtual terminal module, intelligent identification module, and live monitoring display module.

[0067] In the embodiments of the present application, the live quality monitoring management module is used for task management and system configuration operation of cloud terminal monitoring live channels, and querying and displaying live quality monitoring reports; the cloud terminal management module is used for executing monitoring tasks issued by the portal system, including resource management and distribution of virtual terminals, virtual terminal state monitoring, and virtual terminal operation script command issuing; the virtual terminal module is used for creating an Android image of a virtual machine set-top box (in actual application, other systems can be set), and each virtual terminal instance is corresponded to a specified CDN node to obtain streaming media services according to script commands, automatically intercepts live pictures, and generates terminal play logs; the intelligent identification module is used for image recognition of live fault pictures and analysis of play logs, collects live screenshots of virtual machine set-top boxes in batches, identifies play faults such as black screen and flower screen by using visual AI models, and analyzes live quality problems such as play lag and flow interruption in combination with terminal play logs; and the live monitoring display module is used for centralized display of live monitoring pictures of virtual machine set-top boxes, flexible combination of play picture arrangement styles, and real-time display of alarm pop-up windows of live faults.

[0068] According to the above-mentioned virtual terminal-based visual monitoring system visual channel monitoring system, the visual monitoring process is as shown in the following steps. Figure 4

[0069] S401: A user logs in to the virtual terminal-based visual channel monitoring system, and configures channels and CDN nodes to be monitored in the management portal of the live quality monitoring management module, and presets monitoring tasks and alarm rules;

[0070] S402: In the cloud terminal management platform of the cloud terminal management module, according to the monitoring rules set in the management portal, virtual machine set-top box instances are generated by using the virtual terminal module, and monitoring tasks and play addresses of live code streams are issued to each virtual machine set-top box instance in batches;

[0071] S403: Each virtual machine set-top box is distributed to a specified CDN node to obtain live streams, and play logs are collected by the client APK of the virtual machine set-top box;

[0072] S404: Based on the script of the monitoring task, the virtual machine set-top box in the virtual terminal module periodically takes screenshots and uploads the play to the intelligent identification module;

[0073] ​S405: The intelligent identification module intelligently identifies the received monitoring data, and performs comprehensive comparison and analysis based on the picture QoE (full name: Quality of Experience, which is a concept in the communication field, refers to the subjective feeling of users on the quality and performance of devices, networks and systems, applications or services) experience quality and Internet Protocol (Internet Protocol, IP) layer Key Performance Indicators (Key Performance Indicators, KPI) quality indicators, generates an alarm notification according to the alarm rules preset by the portal, and sends the alarm notification to the live monitoring display module to generate a quality report corresponding to the monitoring task;

[0074] S406: The live monitoring display module collects the video stream of the virtual machine set-top box for merging, provides a graphical display page, and presents the alarm notification of the intelligent identification module in the form of a pop-up window on the monitoring screen.

[0075] In actual application, as shown in Figure 5 The specific process of merging the video streams played by multiple virtual terminals to obtain a monitoring video can include the following steps:

[0076] Step 501: AIC 1AIC 2AIC N. In this step, the live display module can establish a websocket (a network transmission protocol that can communicate full-duplex on a single Transmission Control Protocol (TCP) connection) connection with the client; the client sends the container information corresponding to the virtual terminal that needs to be connected; the media service establishes multiple threads, respectively connects the corresponding virtual terminal container and provides websocket service, and receives the video stream information of multiple virtual terminals.

[0077] Step 502: Media Unpacking 1Media Unpacking 2Media Unpacking N.

[0078] Step 503: H264 Media Decoding H264 Media Decoding H264 Media Decoding. In steps 502 and 503, the media service can detect the stream information of each virtual terminal, unpack and decode the H.264 (an international standard format of moving picture coding method jointly formulated by ITU-T and ISO / IEC) information, and restore it to the original YUV (a color coding method) data.

[0079] Step 504: Filter Conversion. In this step, the media service can send multiple YUV data to a single filter processing module. The filter supports multiple input and single output, processes the synthesis of video streams, and synthesizes a single YUV stream.

[0080] Step 505: H264 encoding. In this step, the media service can encode the single YUV stream into an H264 bare stream.

[0081] Step 506: Media packaging. In this step, the media service can add a private protocol header and package it into a complete video message.

[0082] Step 507: Push user. In this step, the packaged video message can be pushed to the client. Then the client completes the stream receiving and decodes and renders to realize the stress test monitoring.

[0083] The embodiment of the application provides a visual monitoring method based on a virtual terminal. An execution subject can be a visual monitoring method based on a virtual terminal device. In the embodiment of the application, the visual monitoring method based on a virtual terminal device is taken as an example to illustrate the visual monitoring method based on a virtual terminal device provided by the embodiment of the application.

[0084] Figure 6 A structural schematic diagram of the visual monitoring method based on a virtual terminal device provided by an example embodiment of the application is shown. The visual monitoring method based on a virtual terminal device can implement all or part of the content in the example embodiment shown in the figure. The visual monitoring method based on a virtual terminal device includes a playing module 601, an identifying module 602, an obtaining module 603 and a displaying module 604. Figure 1

[0085] In the embodiment of the application, the playing module 601 is configured to receive and play a target video corresponding to a video stream of a to-be-monitored channel by using a virtual terminal. The identifying module 602 is configured to identify whether a monitoring index of the played target video is abnormal. The obtaining module 603 is configured to obtain an alarm notification corresponding to the target video in a case where it is identified that the monitoring index of the played target video is abnormal. The displaying module 604 is configured to display the target video and the alarm notification.

[0086] In an optional implementation, when the playing module 601 is configured to receive and play the target video corresponding to the video stream of the to-be-monitored channel by using the virtual terminal, the playing module 601 is specifically configured to:

[0087] obtain monitoring task configuration information corresponding to the virtual terminal, wherein the monitoring task configuration information includes playing address information of the to-be-monitored channel.

[0088] access a content distribution network (CDN) node corresponding to the playing address information by using the virtual terminal, and receive and play the target video corresponding to the video stream of the to-be-monitored channel. ​

[0089] In an optional implementation, the monitoring index includes a play quality; the identification module 602, in the process of identifying whether the monitoring index of the target video played is abnormal, specifically identifies whether the play quality of the target video played is abnormal by:

[0090] collecting a play log of the target video through the virtual terminal;

[0091] identifying whether the play quality of the target video played is abnormal by analyzing the play log.

[0092] In an optional implementation, the monitoring index includes a play content; the identification module 602, in the process of identifying whether the monitoring index of the target video played is abnormal, specifically identifies whether the play content of the target video played is abnormal by:

[0093] obtaining at least one frame of video image of the target video;

[0094] identifying whether the play content of the target video played is abnormal by identifying the content of at least one frame of the video image.

[0095] In an optional implementation, the identification module 602, in the process of identifying whether the play content of the target video played is abnormal by identifying the content of at least one frame of the video image, specifically determines whether the video image is a first abnormal video frame or a first normal video frame by:

[0096] classifying the video image by using a trained classification model;

[0097] determining the first normal video frame as a second abnormal video frame or a second normal video frame by performing a similarity comparison on the first normal video frame by using a reference picture;

[0098] determining the second abnormal video frame as a third abnormal video frame or a third normal video frame by calculating a difference value of the second abnormal video frame and identifying whether a difference region of the second abnormal video frame is a solid color;

[0099] determining the third abnormal video frame as a fourth abnormal video frame or a fourth normal video frame by identifying whether the third abnormal video frame contains an abnormal character;

[0100] determining whether the play content of the target video played is abnormal according to the first abnormal video frame and the fourth abnormal video frame.

[0101] In an optional implementation, the display module 604, in the process of displaying the target video and the alarm notification, specifically displays the target video and the alarm notification by:

[0102] The video streams played by the plurality of virtual terminals are merged to obtain a monitoring video;

[0103] The monitoring video is displayed on a monitoring screen, and the alarm notification is displayed on the monitoring screen.

[0104] The visualization monitoring method and device based on a virtual terminal in the embodiments of the present application can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted electronic device, a Mobile Internet Device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), and can also be a server, a Network Attached Storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, and the like. The embodiments of the present application are not limited in this regard.

[0105] The visualization monitoring method and device based on a virtual terminal in the embodiments of the present application can be a device with an operating system. The operating system can be an Android operating system, an ios operating system, or other possible operating systems, and the embodiments of the present application are not limited in this regard.

[0106] The visualization monitoring method and device based on a virtual terminal provided in the embodiments of the present application can achieve the following advantages. Figure 1 The processes implemented by the method embodiments are not repeated here.

[0107] Optionally, as shown in Figure 7 The embodiments of the present application further provide an electronic device 700, which includes a processor 701 and a memory 702. The memory 702 has a program or instructions stored thereon, which can be run on the processor 701. When the program or instructions are executed by the processor 701, the steps of the visualization monitoring method based on a virtual terminal are implemented, and the same technical effects are achieved. The processes are not repeated here.

[0108] It should be noted that the electronic device in the embodiments of the present application includes the mobile electronic device and the non-mobile electronic device described above.

[0109] Figure 8 A structural block diagram of another electronic device 800 is shown according to an example embodiment of the present application. The electronic device 800 can be implemented as a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart watch, a television, and the like. The electronic device 800 can also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal, and other names.

[0110] Generally, the electronic device 800 includes a processor 801 and a memory 802.

[0111] The processor 801 can include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 801 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), a PLA (Programmable Logic Array). The processor 801 can also include a main processor and a coprocessor, the main processor being a processor for processing data in an awake state, also referred to as a CPU (Central Processing Unit), and the coprocessor being a low-power processor for processing data in a standby state. In some embodiments, the processor 801 can be integrated with a GPU (Graphics Processing Unit) for rendering and drawing the content to be displayed by the display screen. In some embodiments, the processor 801 can also include an AI (Artificial Intelligence) processor for processing machine learning related computing operations.

[0112] The memory 802 can include one or more computer-readable storage media, which can be non-transitory. The memory 802 can also include a high-speed random access memory, and a non-volatile memory such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 802 is used to store at least one instruction for being executed by the processor 801 to implement all or part of the steps of the virtual terminal based visualization monitoring method according to the method embodiments of the present application.

[0113] In some embodiments, the electronic device 800 may optionally include a peripheral device interface 803 and at least one peripheral device. The processor 801, memory 802, and peripheral device interface 803 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 803 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 804, a display screen 805, a camera assembly 806, an audio circuit 807, and a power supply 808.

[0114] In some embodiments, the electronic device 800 further includes one or more sensors 809. The one or more sensors 809 include, but are not limited to, an accelerometer 810, a gyroscope 811, a pressure sensor 812, an optical sensor 813, and a proximity sensor 814.

[0115] Those skilled in the art will understand that Figure 8 The structure shown does not constitute a limitation on the electronic device 800, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0116] This application also provides a computer-readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described visualization monitoring method based on a virtual terminal and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0117] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0118] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-mentioned visualization monitoring method based on virtual terminals, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0119] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0120] This application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, which, when executed by a computer, implement the steps of the above-described visualization monitoring method based on a virtual terminal and achieve the same technical effect. To avoid repetition, these will not be described again here.

[0121] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the application being indicated by the following claims.

[0122] It is to be understood that the application is not limited to the precise construction herein disclosed and shown in the drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application. The scope of the application is limited only by the claims that follow.

Claims

1. A visualization monitoring method based on a virtual terminal, characterized in that, include: A virtual terminal is used to receive and play the target video corresponding to the video stream of the channel to be monitored. Identify whether there are any abnormalities in the monitoring indicators of the target video being played; If an anomaly is detected in the monitoring metrics of the target video being played, an alarm notification corresponding to the target video is obtained; Display the target video and the alarm notification; The monitoring metrics include the content being played; The process of identifying whether the monitoring metrics of the target video being played are abnormal includes: Obtain at least one frame of the target video image; The video image is classified by using a trained classification model, and the video image is determined to be either the first abnormal video frame or the first normal video frame based on whether there is an abnormal scene in the video image. By comparing the similarity of the first normal video frame with a reference image, the first normal video frame is determined to be a second abnormal video frame or a second normal video frame based on whether there is an abnormal scene in the first normal video frame. By calculating the difference value of the second abnormal video frame and identifying whether the difference region of the second abnormal video frame is a solid color, the second abnormal video frame is determined to be a third abnormal video frame or a third normal video frame. By identifying whether there are abnormal characters in the third abnormal video frame, it is determined whether the third abnormal video frame is the fourth abnormal video frame or the fourth normal video frame. Based on the first abnormal video frame and the fourth abnormal video frame, it is determined whether the playback content of the target video is abnormal.

2. The method according to claim 1, characterized in that, The step of receiving and playing the target video corresponding to the video stream of the channel to be monitored using a virtual terminal includes: Obtain the monitoring task configuration information corresponding to the virtual terminal, wherein the monitoring task configuration information includes the playback address information of the channel to be monitored; By using the virtual terminal to access the content delivery network (CDN) node corresponding to the playback address information, the target video corresponding to the video stream of the channel to be monitored can be received and played.

3. The method according to claim 1, characterized in that, The monitoring metrics include playback quality; identifying whether the monitoring metrics for the target video being played are abnormal includes: The playback logs of the target video are collected through the virtual terminal; By analyzing the playback logs, it can be determined whether there are any abnormalities in the playback quality of the target video being played.

4. The method according to claim 1, characterized in that, The display of the target video and the alarm notification includes: The video streams played by multiple virtual terminals are merged to obtain the surveillance video; The monitoring video is displayed on the monitoring screen, and the alarm notification is also displayed on the monitoring screen.

5. A visualization monitoring device based on a virtual terminal, characterized in that, include: The playback module is used to receive and play the target video corresponding to the video stream of the channel to be monitored using a virtual terminal. The identification module is used to identify whether there are any abnormalities in the monitoring indicators of the target video being played. The acquisition module is used to acquire the alarm notification corresponding to the target video when it is found that the monitoring indicators of the target video being played are abnormal. The display module is used to display the target video and the alarm notification; The monitoring metrics include the content being played; The identification module identifies whether there are any abnormalities in the monitoring indicators of the target video being played, including: Obtain at least one frame of the target video image; The video image is classified by using a trained classification model, and the video image is determined to be either the first abnormal video frame or the first normal video frame based on whether there is an abnormal scene in the video image. By comparing the similarity of the first normal video frame with a reference image, the first normal video frame is determined to be a second abnormal video frame or a second normal video frame based on whether there is an abnormal scene in the first normal video frame. By calculating the difference value of the second abnormal video frame and identifying whether the difference region of the second abnormal video frame is a solid color, the second abnormal video frame is determined to be a third abnormal video frame or a third normal video frame. By identifying whether there are abnormal characters in the third abnormal video frame, it is determined whether the third abnormal video frame is the fourth abnormal video frame or the fourth normal video frame. Based on the first abnormal video frame and the fourth abnormal video frame, it is determined whether the playback content of the target video is abnormal.

6. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing programs or instructions that can run on the processor, the programs or instructions being executed by the processor to implement the steps of the virtual terminal-based visualization monitoring method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the virtual terminal-based visualization monitoring method as described in any one of claims 1 to 4.

8. A computer program product, characterized in that, The computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions that, when executed by a computer, cause the computer to perform the steps of the virtual terminal-based visualization monitoring method as described in any one of claims 1 to 4.

Images