Visual monitoring method and device based on virtual terminal and electronic equipment

Abstract

The invention discloses a visual monitoring method and device based on a virtual terminal and electronic equipment, and belongs to the technical field of personal family IPTV (Internet Protocol Television). The method comprises the following steps: receiving and playing a target video corresponding to a video stream of a to-be-monitored channel by adopting a virtual terminal; identifying whether the monitoring index of the played target video is abnormal or not; when it is identified that the monitoring index of the played target video is abnormal, obtaining an alarm notification corresponding to the target video; and displaying the target video and the alarm notification. In this way, the cost can be reduced.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of personal home IPTV technology, and in particular to a visual monitoring method, device and electronic device based on a virtual terminal. Background Art

[0002] With the development of Internet technology and the increasing demand for home video entertainment, interactive Internet Protocol TV (IPTV) / OTT video platforms, due to their combination with the Internet and the integration of related resources, can provide users with more multimedia service functions and resource updates, and are gradually becoming the mainstream of the market.

[0003] In the related technology, the traditional manual monitoring mode is generally used to monitor the IPTV / OTT video platform. This solution is based on real terminals, and the hardware investment and maintenance costs of the monitoring equipment are high. Summary of the invention

[0004] The embodiments of the present application provide a virtual terminal-based visual monitoring method, device, and electronic device, which can solve the problem of excessively high costs.

[0005] In order to solve the above technical problems, this application is implemented as follows: In a first aspect, an embodiment of the present application provides a visual monitoring method based on a virtual terminal, the method comprising: using a virtual terminal to receive and play a target video corresponding to a video stream of a channel to be monitored; identifying whether there are any abnormalities in monitoring indicators of the target video being played; when it is identified that there are any abnormalities in monitoring indicators of the target video being played, obtaining an alarm notification corresponding to the target video; and displaying the target video and the alarm notification.

[0006] In the second aspect, an embodiment of the present application provides a visual monitoring device based on a virtual terminal, which includes: a playback module, which is used to use a virtual terminal to receive and play a target video corresponding to a video stream of a channel to be monitored; an identification module, which is used to identify whether there is an abnormality in the monitoring indicators of the target video being played; an acquisition module, which is used to obtain an alarm notification corresponding to the target video when it is identified that there is an abnormality in the monitoring indicators of the target video being played; and a display module, which is used to display the target video and the alarm notification.

[0007] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect above are implemented.

[0008] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect above are implemented.

[0009] In the seventh aspect, an embodiment of the present application provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the steps of the method described in the first aspect above.

[0010] The technical solution provided by this application may have the following beneficial effects: In the embodiment of the present application, a virtual terminal can be used to receive and play a target video corresponding to a video stream of a channel to be monitored; identify whether the monitoring index of the played target video is abnormal; obtain an alarm notification corresponding to the target video when an abnormality is identified in the monitoring index of the played target video; and display the target video and the alarm notification. In the above manner, costs can be reduced.

[0011] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0013] Figure 1 A flow chart of a virtual terminal-based visual monitoring method provided in an embodiment of the present application is shown; Figure 2a A schematic diagram of a process of an image recognition method provided in an embodiment of the present application is shown; Figure 2b A schematic diagram showing a flow chart of another image recognition method provided in an embodiment of the present application is shown; Figure 3 A schematic diagram of a virtual terminal-based visual monitoring system provided in an embodiment of the present application is shown; Figure 4 A schematic diagram of another virtual terminal-based visual monitoring method provided in an embodiment of the present application is shown; Figure 5 A schematic diagram of merging video streams provided in an embodiment of the present application is shown; Figure 6 A schematic diagram of the structure of a virtual terminal-based visual monitoring device provided in an embodiment of the present application is shown; Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown; Figure 8 A schematic structural diagram of another electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0014] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0015] With the development of Internet technology and the increasing demand for home video entertainment, traditional cable television is gradually withdrawing from the current home video entertainment market. IPTV / OTT (over the top, which refers to providing various application services to users through the Internet) video platform, due to its combination with the Internet and the integration of related resources, can provide users with more multimedia service functions and resource updates, and is gradually becoming the mainstream of the market.

[0016] IPTV, as interactive network television, is an emerging media technology that integrates the Internet, multimedia, and communication technologies, uses broadband networks as a medium, and provides home users with a variety of interactive services including digital television. It has a series of functions such as live broadcast, on-demand, time-shifting, and interactive viewing, and realizes substantive interaction between media providers and media consumers.

[0017] OTT, as Internet TV, is an Internet TV content service platform and integrated service platform that can provide users with a variety of application services through the Internet.

[0018] In the related technologies, on the one hand, video surveillance is still based on real terminals; on the other hand, although IPTV / OTT video platforms have made great technological breakthroughs compared to cable TV, the operation and maintenance of IPTV / OTT video platforms also have the following defects: 1. The hardware investment and maintenance cost of set-top box hardware, content delivery network (CDN) stream monitoring equipment, and other monitoring equipment are high.

[0019] 2. IPTV / OTT video resources have high compliance requirements and are difficult to be universal. Monitoring and quality judgment are complex, deployment is cumbersome, monitoring is complex, the upgrade cycle is long, and the workload of manual operation and maintenance has increased dramatically. In actual applications, with the rapid growth of the user scale, the corresponding user data has also exploded, so the multiple peripheral systems and business chains involved have caused a huge increase in the workload of manual operation and maintenance; since the IPTV / OTT video platform includes the integration of diverse video resources, there are huge challenges in the review process to ensure resource compliance. Monitoring and quality judgment for hundreds of live channels are complex, and the related technology relies on the combination of multiple set-top boxes and large screens for display, which has the problems of cumbersome deployment and complex monitoring; since the version of the IPTV / OTT video platform needs to be tested and verified for the CDN nodes of the entire network, there is a defect of a long upgrade cycle; since the IPTV / OTT video platform is based on the Internet to realize data interaction mode and end-to-end video playback, combined with the special attributes of large-screen content, multiple means are required to ensure content security, making the requirements for video content compliance too high and difficult to be universal.

[0020] Figure 1 FIG. 1 is a flow chart of a virtual terminal-based visual monitoring method provided by an exemplary embodiment of the present application. The method 100 can be executed by an electronic device, which can be a terminal device such as a computer. Figure 1 As shown, the method mainly includes the following steps: S101: Using a virtual terminal to receive and play a target video corresponding to a video stream of a channel to be monitored.

[0021] In the embodiment of the present application, a virtual terminal can be used to receive and play the target video corresponding to the video stream of the channel to be monitored. The channel to be monitored can be determined by simulating the playing behavior of the set-top box user. In practical applications, the channel to be monitored can be predicted according to the trained model, and can also be reproduced according to the playing behavior of the actual user, and the embodiment of the present application is not specifically limited. The purpose of simulating user playback is to better cover more possibilities. The virtual terminal can be a virtual set-top box, and the target video corresponding to the video stream of the channel to be monitored can be received and played by the virtual set-top box. In this way, there is no need for a physical set-top box, which can save costs.

[0022] In an optional implementation, the step of using a virtual terminal to receive and play a target video corresponding to a video stream of a channel to be monitored may include the following steps: Step 1, obtaining the monitoring task configuration information corresponding to the virtual terminal, wherein the monitoring task configuration information includes the playback address information of the channel to be monitored; in an embodiment of the present application, the monitoring task configuration information corresponding to the virtual terminal obtained may include the playback address information of the channel to be monitored, and may also include the monitoring task. The playback address information of the channel to be monitored may include the playback address of the channel to be monitored and the information of the CDN node to be monitored, and the CDN node may be manually specified. In an embodiment of the present application, the service quality of the CDN node in the specified area may be monitored in real time.

[0023] Step 2, by using the virtual terminal to access the content distribution network CDN node corresponding to the playback address information, receiving and playing the target video corresponding to the video stream of the channel to be monitored. In an embodiment of the present application, a virtual terminal can be used to access the corresponding content distribution network CDN node to obtain the video stream of the channel to be monitored, and then the target video corresponding to the video stream of the channel to be monitored is played in the background.

[0024] S102: Identify whether there is any abnormality in the monitoring indicators of the target video being played.

[0025] In the embodiment of the present application, it is possible to identify whether there are any abnormalities in the monitoring indicators of the target video being played, thereby realizing live broadcast quality and content security monitoring, and quickly locating faults such as live broadcast freezes, interruptions, audio and video asynchrony, and live broadcast content tampering. In practical applications, the monitoring indicators can be determined based on the QoE (Quality of Experience, a concept in the field of communications, which refers to the user's subjective perception of the quality and performance of equipment, networks and systems, applications or services) experience quality of the picture and the key performance indicators (Key Performance Indicators, KPI) quality indicators at the Internet Protocol (IP) level.

[0026] In an optional implementation, the monitoring indicator includes playback quality; and the identifying whether the monitoring indicator of the target video being played is abnormal includes: Collecting the playback log of the target video through the virtual terminal; By analyzing the playback log, it is identified whether there is an abnormality in the playback quality of the target video being played.

[0027] In an embodiment of the present application, the playback log of the target video can be collected through the client Android application package (AndroidPackage, APK) of the virtual top box; then, by analyzing the playback log, it is identified whether the playback quality of the target video is abnormal, that is, whether there is poor live broadcast quality, such as mosaic, freeze, black screen, etc. In practical applications, a visual artificial intelligence (AI) model can be used to identify playback failures. The visual AI model has the ability of machine vision (Computational Vision, CV) image recognition. By constructing samples of common live broadcast failures and poor playback quality pictures, the training corpus of the image recognition model is collected, and the defective picture annotation is learned. At the same time, the CV small model capability library is released in stages, and the recognition range and recognition accuracy are improved through the continuous iteration and upgrading of the visual AI model. Among them, the defective pictures used for model training include but are not limited to: large black screens, default filled posters, and flower screen areas. No manual recognition is required, which can not only improve efficiency and reduce costs, but also improve accuracy.

[0028] In an optional implementation, the monitoring indicator includes playback content; and the identifying whether the monitoring indicator of the target video being played is abnormal includes: Acquire a video image of at least one frame of the target video; By identifying the content of at least one frame of the video image, it is identified whether there is any abnormality in the playback content of the target video.

[0029] In an embodiment of the present application, at least one frame of video image of the target video can be obtained according to the monitoring task; then, by identifying the content of the at least one frame of the video image, it is identified whether there is any abnormality in the playback content of the target video, such as whether there is any erroneous recognition, station logo recognition, text recognition, etc., and the picture quality can also be analyzed.

[0030] In an optional implementation, the content of at least one frame of the video image is identified to identify whether the playback content of the target video is abnormal, such as Figure 2a As shown, the following steps may be included: S201: Determine whether the video image is the first abnormal video frame or the first normal video frame by classifying the video image using a trained classification model; in an embodiment of the present application, at least one frame of video image can be obtained, and then a preliminarily classified model can be used to determine whether the video image is the first abnormal video frame or the first normal video frame. The classification model can quickly identify relatively obvious abnormal scenes, such as scenes where a large area in the picture displays an abnormal pop-up box, an abnormal error code, an abnormal loading of a poster, and so on. By identifying the video image through the classification model, if there is a relatively obvious abnormal scene in the video image, it can be determined that the video image is the first abnormal video frame; if there is no relatively obvious abnormal scene in the video image, it can be determined that the video image is the first normal video frame.

[0031] S202: By using a reference image to perform a similarity comparison on the first normal video frame, determine that the first normal video frame is the second abnormal video frame or the second normal video frame; in the embodiment of the present application, if there is an error in a small area of ​​the video image, such as text description, garbled language, etc., it may be classified as a normal scene when classified by the classification model, that is, determine that the video image is the first normal video frame. At this time, a secondary screening is required to determine that the first normal video frame is the second abnormal video frame or the second normal video frame by comparing the similarity with the reference image. In the case where the result of the similarity comparison of the first normal video frame using the reference image is greater than the set threshold, it can be determined that the first normal video frame is the second normal video frame, indicating that it is a normal scene; in the case where the result of the similarity comparison of the first normal video frame using the reference image is less than the set threshold, it can be determined that the first normal video frame is the second abnormal video frame, indicating that it is an abnormal scene. By performing a secondary analysis on the first normal video frame, the recognition accuracy of the smaller difference area can be improved through the similarity ratio result, further ensuring the accuracy of data analysis.

[0032] S203: Determine whether the second abnormal video frame is the third abnormal video frame or the third normal video frame by calculating the difference value of the second abnormal video frame and identifying whether the difference area of ​​the second abnormal video frame is a pure color; in this embodiment of the present application, for the abnormal scene picture (second abnormal video frame) screened out for the second time by S201 and S202, use the opencv (an open source computer vision and machine learning software library) related interface to calculate the difference value, mark the difference area, and determine whether the difference area is a pure color. If it is a pure color, classify it into a normal scene, and determine that the second abnormal video frame is the third normal video frame. If it is not a pure color, classify it into an abnormal scene, and determine that the second abnormal video frame is the third abnormal video frame.

[0033] S204: By identifying whether the third abnormal video frame contains abnormal characters, determine whether the third abnormal video frame is the fourth abnormal video frame or the fourth normal video frame; in the embodiment of the present application, it is possible to identify whether the third abnormal video frame contains abnormal characters. In practical applications, optical character recognition (OCR) can be used to extract text from the non-pure color difference area in the third abnormal video frame to determine whether it contains abnormal characters. If it does not contain abnormal characters, it can be classified into a normal scene, that is, the third abnormal video frame is determined to be the fourth normal video frame. If it contains abnormal characters, it can be classified into an abnormal scene, that is, the third abnormal video frame is determined to be the fourth abnormal video frame.

[0034] S205: Determine whether the playback content of the target video is abnormal based on the first abnormal video frame and the fourth abnormal video frame. In the embodiment of the present application, the first abnormal video frame and the fourth abnormal video frame are video images with identified abnormalities, which can be used to indicate that the playback content of the target video is abnormal, and can further indicate what aspects of the abnormality exist in the playback content of the target video.

[0035] In the embodiment of the present application, the above identification process can also be performed as follows: Figure 2b As shown. In the process of identifying video images, the "coarse recognition + fine analysis" mode can not only ensure the analysis accuracy, but also reduce the occupation of computing resources. In S201, the abnormal images with obvious differences are divided through coarse recognition, and then fine recognition is performed based on the results of coarse recognition (further analysis of the first normal video frame). Compared with the related art of blindly inputting all images into the difference analysis model, the embodiment of the present application can effectively reduce the pressure of model operation and improve the rate of data analysis.

[0036] In another optional implementation, the video image can be distinguished according to whether there is a ready-made abnormal scene image during the image recognition process. There are mainly two scenarios, one with a reference image and the other without a reference image. When the facing video image is a reference image, the video image and the reference image can be directly compared for similarity to obtain a similarity, and a similarity threshold is used to determine whether it is a normal scene or an abnormal scene. In practical applications, the similarity comparison algorithm provided by opencv can be used. When the video image is a reference-free image, the image classification is mainly carried out through the model. At present, there are relatively mature models for image classification, which are basically based on convolutional neural network models, such as VGG16 (a powerful pre-trained model that can be used to identify similarities between images), YOLOV3 (a real-time object detection algorithm that can identify specific objects in videos, real-time sources or images), Single Shot MultiBox Detector (SSD), etc. In practical applications, an image classification model can be obtained through moderate training based on the VGG16 model, and the VGG16 model can be customized through live fault samples for transfer learning, that is, the trained model is used with a small number of pictures for customized training. After completing transfer learning, a model based on VGG16 is trained and saved as a file called classify.keras, which can be directly loaded for subsequent use. After loading the model, the model can be used to complete fault judgment of the acquired video images.

[0037] In the embodiment of the present application, the above two image recognition methods are in parallel relationship. In practical applications, you can choose between the above two image recognition methods, or you can use them simultaneously. The embodiment of the present application does not make any specific limitations.

[0038] In an optional implementation, it is also possible to accurately and efficiently identify whether there are risky image contents such as sensitive, advertisements, logos (LOGOtype, Logo) watermarks, etc. in the video image based on deep learning image detection technology. The detection range of video image recognition based on deep learning image detection technology is relatively larger than the detection range of step S204 above. In practical applications, it can be used according to actual conditions, and both methods can be used for recognition at the same time, which is not specifically limited in the embodiments of this application.

[0039] S103: When it is identified that the monitoring indicators of the target video being played are abnormal, an alarm notification corresponding to the target video is obtained.

[0040] In an embodiment of the present application, when it is identified that the monitoring indicators of the target video being played are abnormal, an alarm notification corresponding to the target video can be obtained according to the preset alarm rules. The alarm rule can be that the playback quality indicator reaches a preset threshold. In practical applications, a quality report corresponding to the monitoring task can also be generated. Not only can it perform synchronous real-time identification and analysis of the monitored channels to improve monitoring efficiency, but it can also meet real-time requirements and help ensure the quality of video services and improve operational efficiency.

[0041] S104: Display the target video and the alarm notification.

[0042] In an embodiment of the present application, the target video and the alarm notification can be displayed, thereby assisting the user to deal with non-compliant content in a timely manner and ensure the compliance of the video playback content.

[0043] In an optional implementation, the displaying of the target video and the alarm notification includes: Combining the video streams played by the multiple virtual terminals to obtain a surveillance video; The monitoring video is displayed on a monitoring screen, and the alarm notification is displayed on the monitoring screen.

[0044] In the embodiment of the present application, multiple channels to be monitored can be monitored simultaneously, and then the target videos corresponding to the multiple channels to be monitored can be obtained. At this time, the video streams of multiple virtual terminals can be collected and merged to obtain the monitoring video, and then the monitoring video is displayed on the monitoring screen, and the alarm notification corresponding to the target video is also displayed on the monitoring screen. In actual applications, the alarm notification can be presented on the monitoring screen in the form of a pop-up window, which can be a modal pop-up window, a non-modal pop-up window, or other forms other than a pop-up window, which is not specifically limited in the embodiment of the present application.

[0045] In the embodiment of the present application, a virtual terminal can be used to receive and play the target video corresponding to the video stream of the channel to be monitored; identify whether the monitoring indicators of the target video being played are abnormal; when the monitoring indicators of the target video being played are abnormal, obtain the alarm notification corresponding to the target video; display the target video and the alarm notification. Compared with related live monitoring methods, the embodiment of the present application does not need to deploy a large number of physical terminals, which greatly reduces the cost of hardware equipment investment and manual monitoring.

[0046] Figure 3 A schematic diagram of a virtual terminal-based visual monitoring system provided in an embodiment of the present application is shown. The virtual terminal visual monitoring system may include the following modules: a live broadcast quality monitoring management module, a cloud terminal management module, a virtual terminal module, an intelligent identification module, and a live broadcast monitoring display module.

[0047] In an embodiment of the present application, a live broadcast quality monitoring management module is used for task management and system configuration operations for monitoring live broadcast channels for cloud terminals, and for querying and displaying live broadcast quality monitoring reports; a cloud terminal management module is used to execute monitoring tasks issued by a portal system, including resource management and allocation of virtual terminals, virtual terminal status monitoring, and issuance of virtual terminal operation script commands; a virtual terminal module is used to create an Android image of a virtual top box (in actual applications, it can be set to other systems), and each virtual terminal instance corresponds to a designated CDN node according to a script command to obtain streaming media services, automatically capture live broadcast images, and generate terminal playback logs; an intelligent recognition module is used for image recognition and playback log analysis of live broadcast fault images, by batch collecting live broadcast screenshots of virtual top boxes, using visual AI models to identify playback faults such as black screens and flowery screens, and combining terminal playback logs to analyze playback freezes, interruptions, and other poor live broadcast quality problems; a live broadcast monitoring and display module is used to centrally display the live broadcast monitoring images of virtual top boxes, flexibly combine playback image layout styles, and display alarm pop-ups of live broadcast faults in real time.

[0048] According to the above-mentioned virtual terminal-based visual monitoring system visual channel monitoring system, the visual monitoring process is performed as follows: Figure 4 As shown, the following steps are included: S401: The user logs in to the visual channel monitoring system based on the virtual terminal, configures the channel and CDN node to be monitored in the management portal of the live broadcast quality monitoring management module, and presets the monitoring tasks and alarm rules; S402: In the cloud terminal management platform of the cloud terminal management module, according to the monitoring rules set in the management portal, a virtual top box instance is generated by using the virtual terminal module, and monitoring tasks and the playback address of the live stream are sent in batches to each virtual top box instance; S403: Allocate each virtual TTB to a designated CDN node, obtain the live stream, and collect the playback log through the client APK of the virtual TTB; S404: Based on the script of the monitoring task, the virtual top box in the virtual terminal module regularly takes screenshots and uploads them to the intelligent recognition module for playback; S405: The intelligent identification module intelligently identifies the received monitoring data, and conducts a comprehensive comparative analysis based on the QoE (Quality of Experience, a concept in the field of communications, which refers to the user's subjective feeling of the quality and performance of equipment, networks and systems, applications or services) of the picture and the key performance indicators (KPI) quality indicators at the Internet Protocol (IP) level. At the same time, according to the alarm rules preset by the portal, an alarm notification is generated, and the alarm notification is sent to the live monitoring display module to generate a quality report corresponding to the monitoring task; S406: The live broadcast monitoring and display module collects the video stream of the virtual set-top box and merges them, provides a graphical display page, and presents the alarm notification of the intelligent recognition module on the monitoring screen in the form of a pop-up window or the like.

[0049] In practical applications, such as Figure 5 As shown, the specific process of combining the video streams played by multiple virtual terminals to obtain the surveillance video may include the following steps: Step 501: AIC 1AIC 2AIC N. In this step, the live display module can establish a websocket (a network transmission protocol that can perform full-duplex communication on a single Transmission Control Protocol (TCP) connection) connection with the client; the client sends the container information corresponding to the virtual terminal to be connected; the media service establishes multiple threads, connects to the corresponding virtual terminal containers respectively and provides websocket services to receive video stream information of multiple virtual terminals.

[0050] Step 502: Media decapsulation 1 Media decapsulation 2 Media decapsulation N.

[0051] Step 503: H264 media decoding H264 media decoding H264 media decoding. In step 502 and step 503, the media service can detect the stream information of each virtual terminal respectively, decapsulate and decode the stream H.264 (an international standard format for a moving image encoding method jointly formulated by the International Telecommunication Standardization Department ITU-T and the International Organization for Standardization ISO / International Electrotechnical Commission IEC that specifies MPEG) information, and restore it to the original YUV (a color encoding method) data.

[0052] Step 504: Filter conversion: In this step, the media service can send multiple channels of YUV data to a single filter processing module. The filter supports multiple inputs and a single output, and synthesizes the video streams into a single YUV stream.

[0053] Step 505: H264 encoding: In this step, the media service can encode the single-channel YUV stream into an H264 raw stream.

[0054] Step 506: Media encapsulation: In this step, the media service can add a private protocol header to encapsulate it into a complete video message.

[0055] Step 507: Pushing the user. In this step, the packaged video message can be pushed to the client. Then the client completes the streaming, decoding and rendering, and realizes the dial-up monitoring.

[0056] The embodiment of the present application provides a visual monitoring method based on a virtual terminal, and the execution subject can be a visual monitoring method device based on a virtual terminal. In the embodiment of the present application, a visual monitoring method device based on a virtual terminal performs a visual monitoring method based on a virtual terminal as an example to illustrate the visual monitoring method device based on a virtual terminal provided by the embodiment of the present application.

[0057] Figure 6 The structure diagram of a virtual terminal-based visual monitoring method and device provided by an exemplary embodiment of the present application is shown. The virtual terminal-based visual monitoring method and device can be implemented as follows: Figure 1 All or part of the content in the illustrated embodiment, the visual monitoring method and device based on the virtual terminal includes: a playing module 601, an identification module 602, an acquisition module 603 and a display module 604.

[0058] In an embodiment of the present application, a playback module 601 is used to use a virtual terminal to receive and play a target video corresponding to a video stream of a channel to be monitored; an identification module 602 is used to identify whether there is an abnormality in monitoring indicators of the target video being played; an acquisition module 603 is used to obtain an alarm notification corresponding to the target video when it is identified that there is an abnormality in the monitoring indicators of the target video being played; and a display module 604 is used to display the target video and the alarm notification.

[0059] In an optional implementation, when the playing module 601 is used to receive and play the target video corresponding to the video stream of the channel to be monitored by using the virtual terminal, it is specifically used to: Acquire monitoring task configuration information corresponding to the virtual terminal, wherein the monitoring task configuration information includes the broadcast address information of the channel to be monitored; The target video corresponding to the video stream of the channel to be monitored is received and played by using the virtual terminal to access the content distribution network CDN node corresponding to the playback address information.

[0060] In an optional implementation, the monitoring indicator includes playback quality; when the identification module 602 is used to identify whether the monitoring indicator of the target video being played is abnormal, it is specifically used to: Collecting the playback log of the target video through the virtual terminal; By analyzing the playback log, it is identified whether there is an abnormality in the playback quality of the target video being played.

[0061] In an optional implementation, the monitoring indicator includes playback content; when the identification module 602 is used to identify whether the monitoring indicator of the target video being played is abnormal, it is specifically used to: Acquire a video image of at least one frame of the target video; By identifying the content of at least one frame of the video image, it is identified whether there is any abnormality in the playback content of the target video.

[0062] In an optional implementation, when the identification module 602 is used to identify whether the playback content of the target video is abnormal by identifying the content of at least one frame of the video image, it is specifically used to: Classifying the video image by using a trained classification model to determine whether the video image is a first abnormal video frame or a first normal video frame; By using a reference picture to perform a similarity comparison on the first normal video frame, determining that the first normal video frame is a second abnormal video frame or a second normal video frame; Determining whether the second abnormal video frame is a third abnormal video frame or a third normal video frame by calculating a difference value of the second abnormal video frame and identifying whether a difference region of the second abnormal video frame is a pure color; By identifying whether the third abnormal video frame has an abnormal character, determining that the third abnormal video frame is a fourth abnormal video frame or a fourth normal video frame; It is determined whether there is any abnormality in the playback content of the target video according to the first abnormal video frame and the fourth abnormal video frame.

[0063] In an optional implementation, when the display module 604 is used to display the target video and the alarm notification, it is specifically used to: Combining the video streams played by the multiple virtual terminals to obtain a surveillance video; The monitoring video is displayed on a monitoring screen, and the alarm notification is displayed on the monitoring screen.

[0064] The visual monitoring method device based on the virtual terminal in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or it can be other devices other than the terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0065] The virtual terminal-based visual monitoring method device in the embodiment of the present application may be a device with an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0066] The visual monitoring method and device based on the virtual terminal provided in the embodiment of the present application can realize Figure 1 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0067] Alternatively, if Figure 7 As shown, an embodiment of the present application also provides an electronic device 700, including a processor 701 and a memory 702, wherein the memory 702 stores programs or instructions that can be executed on the processor 701, and when the program or instructions are executed by the processor 701, the various steps of the above-mentioned virtual terminal-based visual monitoring method are implemented, and the same technical effect can be achieved. To avoid repetition, they are not repeated here.

[0068] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0069] Figure 8FIG. 8 is a block diagram of another electronic device 800 according to an exemplary embodiment of the present application. The electronic device 800 may be implemented as a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart watch, a TV, etc. The electronic device 800 may also be called a user device, a portable terminal, a laptop terminal, a desktop terminal, etc.

[0070] Typically, the electronic device 800 includes a processor 801 and a memory 802 .

[0071] The processor 801 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 801 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 801 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 801 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 801 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0072] The memory 802 may include one or more computer-readable storage media, which may be non-transitory. The memory 802 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 802 is used to store at least one instruction, which is used to be executed by the processor 801 to implement all or part of the steps in the virtual terminal-based visual monitoring method shown in the method embodiment of the present application.

[0073] In some embodiments, the electronic device 800 may further optionally include: a peripheral device interface 803 and at least one peripheral device. The processor 801, the memory 802 and the peripheral device interface 803 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 803 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 804, a display screen 805, a camera assembly 806, an audio circuit 807 and a power supply 808.

[0074] In some embodiments, the electronic device 800 further includes one or more sensors 809 , including but not limited to: an acceleration sensor 810 , a gyroscope sensor 811 , a pressure sensor 812 , an optical sensor 813 , and a proximity sensor 814 .

[0075] Those skilled in the art will understand that Figure 8 The structure shown in the figure does not constitute a limitation on the electronic device 800, and may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0076] An embodiment of the present application also provides a computer-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned virtual terminal-based visual monitoring method are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0077] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0078] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned virtual terminal-based visual monitoring method, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0079] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0080] An embodiment of the present application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the steps of the above-mentioned virtual terminal-based visual monitoring method are implemented, and the same technical effect can be achieved. To avoid repetition, they are not repeated here.

[0081] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0082] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A visual monitoring method based on a virtual terminal, characterized in that: include: Using a virtual terminal to receive and play the target video corresponding to the video stream of the channel to be monitored; Identify whether there is an abnormality in the monitoring indicators of the target video being played; When it is identified that the monitoring indicators of the target video being played are abnormal, an alarm notification corresponding to the target video is obtained; The target video and the alarm notification are displayed.

2. The method according to claim 1, characterized in that The method of using a virtual terminal to receive and play a target video corresponding to a video stream of a channel to be monitored includes: Acquire monitoring task configuration information corresponding to the virtual terminal, wherein the monitoring task configuration information includes the broadcast address information of the channel to be monitored; The target video corresponding to the video stream of the channel to be monitored is received and played by using the virtual terminal to access the content distribution network CDN node corresponding to the playback address information.

3. The method according to claim 1, characterized in that The monitoring indicator includes the playback quality; the identification of whether the monitoring indicator of the target video being played is abnormal includes: Collecting the playback log of the target video through the virtual terminal; By analyzing the playback log, it is identified whether there is an abnormality in the playback quality of the target video being played.

4. The method according to claim 1 or 3, characterized in that: The monitoring indicator includes the playback content; the identification of whether the monitoring indicator of the target video being played is abnormal includes: Acquire a video image of at least one frame of the target video; By identifying the content of at least one frame of the video image, it is identified whether there is any abnormality in the playback content of the target video.

5. The method according to claim 4, characterized in that The step of identifying whether the playback content of the target video is abnormal by identifying the content of at least one frame of the video image includes: Classifying the video image by using a trained classification model to determine whether the video image is a first abnormal video frame or a first normal video frame; By using a reference picture to perform a similarity comparison on the first normal video frame, determining that the first normal video frame is a second abnormal video frame or a second normal video frame; Determining whether the second abnormal video frame is a third abnormal video frame or a third normal video frame by calculating a difference value of the second abnormal video frame and identifying whether a difference region of the second abnormal video frame is a pure color; By identifying whether the third abnormal video frame has an abnormal character, determining that the third abnormal video frame is a fourth abnormal video frame or a fourth normal video frame; It is determined whether there is any abnormality in the playback content of the target video according to the first abnormal video frame and the fourth abnormal video frame.

6. The method according to claim 1, characterized in that The displaying of the target video and the warning notification includes: Combining the video streams played by the multiple virtual terminals to obtain a surveillance video; The monitoring video is displayed on a monitoring screen, and the alarm notification is displayed on the monitoring screen.

7. A visual monitoring device based on a virtual terminal, characterized in that: include: A playback module, used to use a virtual terminal to receive and play a target video corresponding to a video stream of a channel to be monitored; An identification module, used to identify whether the monitoring indicators of the target video being played are abnormal; An acquisition module, configured to acquire an alarm notification corresponding to the target video when an abnormality is detected in the monitoring indicators of the target video being played; A display module is used to display the target video and the alarm notification.

8. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the virtual terminal-based visual monitoring method as described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the virtual terminal-based visual monitoring method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the steps of the virtual terminal-based visual monitoring method as described in any one of claims 1 to 6.

Images