Display device monitoring method, system and computer program product

Abstract

The present disclosure provides a display device monitoring method and system, electronic device, storage medium and computer program product, relates to the technical field of computers, in particular to the technical fields of large language models, image recognition, natural language processing, terminal monitoring, and can be applied to the monitoring scene. The specific implementation scheme is: obtaining running data and multi-modal display data of the display device in the data display process; monitoring the running state of the display device according to the running data to obtain a first monitoring result; using a plurality of monitoring strategies corresponding to the multi-modal display data to process the multi-modal display data to obtain a second monitoring result; in response to the first monitoring result and / or the second monitoring result indicating that there is an abnormal situation in the data display process, determining and executing an abnormal processing task corresponding to the abnormal situation through the display device. The present disclosure reduces the intensity of manual monitoring while improving the safety and reliability of the display process.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, in particular to the technical fields of large language model, image recognition, natural language processing, terminal monitoring, and the like, and more particularly to a monitoring method and system of display equipment, an electronic device, a storage medium, and a computer program product, which can be applied to a monitoring scene. BACKGROUND

[0002] With the continuous development of large language model technology, applications equipped with AIGC (Artificial Intelligence Generated Content) technology have been widely applied in digital exhibition hall design. These applications can not only interpret the information, background and practical value of scientific and technological exhibits through natural language interaction, but also interactively answer the questions of visiting users in a dialogue manner, so that the audience can obtain more personalized, intelligent and interactive exhibition experience, further improving the attraction and experience of digital exhibition. As an important medium for information dissemination and interaction in a scientific and technological exhibition hall, the correctness and fluency of the display content of a digital screen will directly affect the experience of the audience and the effect of information dissemination. However, due to the large number of display equipment terminals in the exhibition hall and the certain degree of freedom of the display content, it is difficult to achieve full-range coverage monitoring by using manual patrol. SUMMARY

[0003] The present disclosure provides a monitoring method and system of display equipment, an electronic device, a storage medium, and a computer program product.

[0004] According to a first aspect, a monitoring method of display equipment is provided, comprising: acquiring running data and multi-modal display data of the display equipment in a data display process; monitoring a running state of the display equipment according to the running data to obtain a first monitoring result; processing the multi-modal display data by using a plurality of monitoring strategies corresponding to the multi-modal display data one-to-one to obtain a second monitoring result; and in response to the first monitoring result and / or the second monitoring result indicating that there is an abnormal situation in the data display process, determining and executing an abnormal processing task corresponding to the abnormal situation by using the display equipment.

[0005] According to a second aspect, a monitoring system of display equipment is provided, comprising: a monitoring client configured to collect running data and multi-modal display data of the display equipment in a data display process; a data processing center configured to monitor a running state of the display equipment according to the running data to obtain a first monitoring result, and process the multi-modal display data by using a plurality of monitoring strategies corresponding to the multi-modal display data one-to-one to obtain a second monitoring result; an abnormal processing center configured to determine an abnormal processing task corresponding to an abnormal situation in response to the first monitoring result and / or the second monitoring result indicating that there is the abnormal situation in the data display process; and the monitoring client is further configured to control the display equipment to execute the abnormal processing task.

[0006] According to a third aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any implementation manner of the first aspect.

[0007] According to a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, and the computer instructions are used to enable a computer to perform the method according to any implementation manner of the first aspect.

[0008] According to a fifth aspect, a computer program product is provided, comprising: a computer program, which, when executed by a processor, implements the method according to any implementation manner of the first aspect.

[0009] According to the technology of the present disclosure, a monitoring method and system of a display device are provided, which perform monitoring in the device dimension and the content dimension based on the running data of the display device and the multi-modal display data, thereby improving the comprehensiveness of the monitoring process; a plurality of monitoring strategies corresponding to the multi-modal display data are adopted to process the multi-modal display data, thereby improving the pertinence and accuracy of the content dimension monitoring process; in response to an abnormal situation existing in the device dimension and / or the content dimension, an abnormal processing task corresponding to the abnormal situation is determined to process the abnormal situation pertinently, thereby reducing the artificial monitoring intensity while improving the safety and reliability of the display process.

[0010] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0011] The accompanying drawings are used to better understand the present scheme and do not limit the present disclosure. Among them:

[0012] Figure 1 is an exemplary system architecture diagram to which an embodiment according to the present disclosure can be applied;

[0013] Figure 2 is a flowchart of an embodiment of a monitoring method of a display device according to the present disclosure;

[0014] Figure 3 is a schematic diagram of an application scenario of a monitoring method of a display device according to the present embodiment;

[0015] Figure 4 is a flowchart of another embodiment of a monitoring method of a display device according to the present disclosure;

[0016] Figure 5 is a structural diagram of one embodiment of a monitoring system of a display device according to the present disclosure;

[0017] Figure 6 is a timing diagram of a monitoring system of a display device according to the present disclosure.

[0018] Figure 7 is a structural diagram of a computer system suitable for implementing embodiments of the present disclosure. DETAILED DESCRIPTION

[0019] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, which are meant to be exemplary and not limiting. Therefore, it should be recognized that many changes and modifications can be made to the embodiments described herein, without departing from the spirit and scope of the present disclosure. Also, for the sake of brevity and clarity, descriptions of well-known functions and constructions are omitted herein.

[0020] In the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information comply with relevant laws and regulations and do not violate public order and good customs.

[0021] Figure 1 An exemplary architecture 100 of a monitoring method and system of a display device to which the present disclosure can be applied is shown.

[0022] As shown in Figure 1 , the system architecture 100 can include terminal devices 101, 102, 103, a network 104 and a server 105. The terminal devices 101, 102, 103 are communicatively connected to form a topology network, and the network 104 is used as a medium to provide communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0023] The terminal devices 101, 102, 103 can be hardware devices or software that support network connection to interact and process data. When the terminal devices 101, 102, 103 are hardware, they can be various electronic devices that support network connection, information acquisition, interaction, display, processing, etc., including but not limited to display devices, digital screens, smartphones, tablet computers, e-book readers, laptop computers and desktop computers, etc. When the terminal devices 101, 102, 103 are software, they can be installed in the above-mentioned electronic devices. They can be implemented as multiple software or software modules for providing distributed services, or as a single software or software module. No specific limitation is made herein.

[0024] Server 105 can be a server that provides various services, such as acquiring operational data and multimodal display data of terminal devices 101, 102, and 103 during the data display process, and serving as a background processing server for monitoring the data display process. As an example, server 105 can be a cloud server.

[0025] It should be noted that a server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules (such as software programs or software modules used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0026] It should also be noted that the monitoring method for the display device provided in the embodiments of this disclosure is generally executed by a server, but the possibility of it being executed by a terminal device, or by the server and terminal device cooperating with each other, is not excluded. Accordingly, the various parts (e.g., various units) included in the monitoring system of the display device can be all set in the server, all set in the terminal device, or set in the server and terminal device respectively.

[0027] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Any number of terminal devices, networks, and servers can be included depending on implementation needs. When the electronic devices on which the monitoring method of the display device runs do not need to transmit data with other electronic devices, the system architecture may only include the electronic devices (e.g., terminal devices or servers) on which the monitoring method of the display device runs.

[0028] Please refer to Figure 2 , Figure 2 A flowchart illustrating a monitoring method for a display device provided in this embodiment of the disclosure. Flowchart 200 includes the following steps:

[0029] Step 201: Obtain the operating data and multimodal display data of the display device during the data display process.

[0030] In this embodiment, the entity executing the monitoring method of the demonstration device (e.g., Figure 1 The server in the system can obtain the operating data and multimodal display data of the display device during the data display process from a remote location or from a local location via a wired network connection or a wireless network connection.

[0031] The display device is a media device for displaying multi-modal display data, supports multiple operating systems (including Windows, Linux, etc. operating systems with graphical interfaces), and is, for example, multiple digital screens in a digital exhibition hall. A monitoring client is deployed in the display device, and the monitoring client can collect running data and multi-modal display data of the display device during data display. The above execution subject can obtain the running data and multi-modal display data collected by the monitoring client. The display device with the locally deployed monitoring client is a monitoring terminal.

[0032] The running data includes static data and dynamic data of the display device. The static data is generally device attribute data, including but not limited to device ID (Identity document, identity), GPU (Graphics Processing Unit, graphics processing unit) model, graphics card driver version, and device asset retirement date. The dynamic data is generally process data of the display device during running, including but not limited to device running time, device temperature, device power consumption, GPU stream processor utilization, GPU video memory utilization, and device running process.

[0033] The multi-modal display data can be various modal display data, including but not limited to text, sound, image, video, etc.

[0034] In the present implementation, in each data collection period, all running data and all multi-modal display data in the period can be collected, or part of the running data and part of the multi-modal display data in the period can be collected. For example, every preset time interval, a collection operation of running data and multi-modal display data is performed, and part of the running data and part of the multi-modal display data in a collection period are obtained.

[0035] For different running data and different modal display data, different preset time intervals can be set to adapt the preset time intervals to different data.

[0036] In some implementations, the monitoring server corresponding to the above execution subject includes a communication gateway.

[0037] The communication gateway is a hub between the monitoring client and the monitoring server, has the ability of bidirectional communication authentication, is responsible for processing and distributing data transmission requests of the monitoring client, and ensures the security and integrity of the data flow.

[0038] Specifically, the deployment and authentication process of the monitoring client is as follows: first, the display device starts the monitoring client, and the monitoring client is continuously run through the daemon process; then, the monitoring client generates a unique device identification code, sends an identity signature authentication request to the communication gateway, and applies for signature authentication. Then, the monitoring gateway receives the identity signature authentication request, and after authentication, stores the authentication mapping relationship between the monitoring client and the display device. Then, the communication gateway sends an authentication pass request (including signature authentication information and global configuration parameters of the monitoring process) to the monitoring client. Finally, the monitoring client receives the authentication pass response of the communication gateway, and successfully establishes a bidirectional data transmission path with the communication gateway.

[0039] In step 202, the running state of the display device is monitored according to the running data, and a first monitoring result is obtained.

[0040] In this embodiment, the above execution subject can monitor the running state of the display device according to the running data, and obtain a first monitoring result.

[0041] As an example, the above execution subject can monitor the running state of the display device based on the prior strategy formed by the experience of the device maintenance personnel, determine whether the running state of the display device is abnormal, and obtain a first monitoring result. The prior strategy is, for example, a judgment rule for various abnormal situations in the device running process.

[0042] As another example, the above execution subject can input the running data into a pre-trained running state monitoring model to determine whether the running state of the display device is abnormal, and obtain a first monitoring result. The running state monitoring model represents the correspondence between the running data and the first monitoring result, and is, for example, a recurrent neural network, a residual network, or the like.

[0043] In step 203, a plurality of monitoring strategies corresponding to the plurality of modal display data are adopted to process the plurality of modal display data, and a second monitoring result is obtained.

[0044] In this embodiment, the above execution subject can adopt a plurality of monitoring strategies corresponding to the plurality of modal display data to process the plurality of modal display data, and obtain a second monitoring result.

[0045] Specifically, for each modal display data in the plurality of modal display data, the modal display data is processed through the monitoring strategy corresponding to the modal display data, and a sub-monitoring result corresponding to the modal display data is obtained; and the second monitoring result is obtained by combining a plurality of sub-monitoring results corresponding to the plurality of modal display data.

[0046] As an example, the monitoring strategy set includes multiple monitoring strategies, including but not limited to a monitoring strategy for determining whether the data is smooth, a monitoring strategy for determining whether the data includes sensitive content, a monitoring strategy for determining whether the modal data is synchronized, and a monitoring strategy for determining whether the data has logicality. For each modal of the multi-modal presentation data, a target monitoring strategy corresponding to the modal of the presentation data is determined from the monitoring strategy set, and the modal of the presentation data is processed by the target monitoring strategy to obtain a sub-monitoring result corresponding to the modal of the presentation data.

[0047] As another example, a plurality of monitoring models corresponding to the multi-modal presentation data are used to process the multi-modal presentation data to obtain a second monitoring result. For each modal of the multi-modal presentation data, the modal of the presentation data is processed by the monitoring model corresponding to the modal of the presentation data to obtain a sub-monitoring result corresponding to the modal of the presentation data. The model corresponding to the modal of the presentation data is used to execute the monitoring strategy corresponding to the modal of the presentation data. Then, the second monitoring result is obtained by combining the plurality of sub-monitoring results corresponding to the multi-modal presentation data.

[0048] Taking the monitoring strategy for determining whether the data includes sensitive content as an example, the above-mentioned execution subject can train the monitoring model using the training sample set corresponding to the detection strategy. Specifically, first, the training sample set is obtained, wherein the training sample in the training sample set includes sample data, a judgment label representing whether the sample data includes sensitive data, and a sensitive data label representing the sensitive data in the case of including sensitive data. Then, a machine learning algorithm is used to train the detection model, with the sample data as the output and the sample data corresponding judgment label and sensitive data label as the expected output.

[0049] It should be noted that the information processing process corresponding to step 202 and the information processing process corresponding to step 203 can be executed in parallel or sequentially, which is not limited here.

[0050] Step 204, in response to the first monitoring result and / or the second monitoring result indicating that the data presentation process has an abnormal situation, determining and executing an abnormal handling task corresponding to the abnormal situation through the presentation device.

[0051] In this embodiment, the above-mentioned execution subject can respond to the first monitoring result and / or the second monitoring result indicating that the data presentation process has an abnormal situation, and determine and execute an abnormal handling task corresponding to the abnormal situation through the presentation device. For example, an abnormal handling center is set in the above-mentioned execution subject, and the abnormal handling center can determine the abnormal handling task corresponding to the abnormal situation.

[0052] As an example, the above execution subject or the electronic device in communication connection with the above execution subject is provided with corresponding relationship data representing the corresponding relationship between the abnormal situation and the abnormal handling task. In response to the first monitoring result and / or the second monitoring result indicating that the data display process has an abnormal situation, the abnormal handling task corresponding to the abnormal situation is determined from the corresponding relationship data, and the abnormal handling task corresponding to the abnormal situation is executed through the display device to solve the abnormal situation.

[0053] As another example, the above execution subject can determine the abnormal handling task corresponding to the abnormal situation based on an abnormal handling model. For example, the abnormal handling model is a deep learning network model, a large language model, etc. with abnormal handling capability.

[0054] In some optional implementations of the present embodiment, an abnormal handling center is provided in the above execution subject, which can execute the above step 204 in the following manner:

[0055] The first step is to determine a plurality of abnormal handling strategies corresponding one-to-one to a plurality of abnormal situations in response to the first monitoring result and / or the second monitoring result indicating that the data display process has a plurality of abnormal situations.

[0056] In the present implementation, for each abnormal situation, its corresponding abnormal handling strategy is set. The abnormal handling strategy is, for example, an operation execution path capable of solving the abnormal situation. According to the order of operations in the operation execution path, the abnormal situation can be solved.

[0057] As an example, for the abnormal situation of the demonstration program crash, its abnormal handling strategy is to reset the system process—>start the demonstration program—>initialize the demonstration program; for the abnormal situation of the demonstration program out of sync, its abnormal handling strategy is to reset the demonstration program refresh; for the abnormal situation of the demonstration program audio channel missing, its abnormal handling strategy is hardware self-check—>sound card driver self-check / installation / update—>start sound card driver—>audio adaptive adjustment.

[0058] For a plurality of abnormal situations in the first monitoring result and / or the second monitoring result, a plurality of abnormal handling strategies corresponding one-to-one to the plurality of abnormal situations are determined.

[0059] The second step is to arrange the plurality of abnormal handling strategies through a strategy arrangement engine to determine the abnormal handling task.

[0060] The strategy arrangement engine can automatically schedule and execute a series of abnormal handling strategies by defining the dependency relationship and execution order between the strategies, thereby improving work efficiency, reducing cost, and improving service quality.

[0061] In this implementation, the policy orchestration engine is used to orchestrate multiple exception handling strategies, for example, to organize, coordinate and integrate multiple operation paths corresponding to multiple exception handling strategies according to certain logic, rules or processes, so as to form a complete and orchestrated operation path, i.e., an exception handling task.

[0062] In this implementation, the intelligent orchestration module is included in the exception handling center, and the first and second steps can be executed by the intelligent orchestration module.

[0063] In this implementation, a specific determination method of the exception handling task is provided, which can cope with complex scenarios in which the data display process includes multiple exception conditions, and improves the processing efficiency and processing capacity of the exception conditions.

[0064] In some optional implementations of the embodiment, the execution subject can execute the step 204 in the following manner:

[0065] In the third step, the orchestration task of the multiple exception handling strategies is sent to the target terminal in response to the failure of the policy orchestration engine to orchestrate the multiple exception handling strategies. In the fourth step, the exception handling task is obtained based on the target terminal.

[0066] For the case where automatic orchestration is not supported or the orchestration process fails to successfully orchestrate, the manual orchestration module is triggered to send the orchestration task of the multiple exception handling strategies to the target terminal, notify the on-duty operation and maintenance personnel to intervene, and uniformly orchestrate the multiple exception handling strategies, so that when the system self-healing capability is insufficient, the potential impact can be reduced through manual intervention.

[0067] In this implementation, the manual orchestration method ensures the orderly processing of the exception conditions, and further improves the security and reliability of the monitoring and broadcasting process.

[0068] Continuing to refer to Figure 3 , Figure 3 is one schematic diagram of the application scenario 300 of the monitoring and broadcasting method of the display device according to the embodiment. The server 301 first obtains the running data and the multi-modal display data of each of the multiple display devices 302 in the data display process. For each display device, the running state of the display device is monitored according to the running data of the display device to obtain a first monitoring and broadcasting result; the multi-modal display data corresponding to the multiple monitoring and broadcasting strategies are used to process the multi-modal display data corresponding to the running device to obtain a second monitoring and broadcasting result; finally, in response to the first monitoring and broadcasting result and / or the second monitoring and broadcasting result indicating that there is an exception condition in the data display process, an exception handling task corresponding to the exception condition is determined and executed by the display device.

[0069] In this embodiment, a monitoring method of a display device is provided. The method performs monitoring in the device dimension and the content dimension based on the operation data of the display device and the multi-modal display data, thereby improving the comprehensiveness of the monitoring process. The multi-modal display data is processed by using multiple monitoring strategies corresponding to the multi-modal display data, thereby improving the pertinence and accuracy of the content dimension monitoring process. In response to an abnormal situation existing in the device dimension and / or the content dimension, an abnormal processing task corresponding to the abnormal situation is determined, so as to process the abnormal situation pertinently, thereby reducing the artificial monitoring intensity and improving the safety and reliability of the display process.

[0070] In some optional implementation of this embodiment, the execution subject can further perform the following operation: determining the target processing node from the data processing center according to the load data of the processing nodes in the data processing center.

[0071] The data processing center includes multiple processing nodes, and each processing node includes an operation data processing module and a display data processing module. As an example, the execution subject can determine the target processing node from the data processing center according to the load data of the processing nodes in the data processing center based on a load balancing strategy.

[0072] In this implementation, the execution subject can perform the step 202 by performing the following operation: performing abnormality detection and risk prediction on the display device according to the operation data by using the operation data processing module in the target processing node, thereby obtaining the first monitoring result.

[0073] The abnormality detection is used to detect whether the display device has an abnormality in the running process in the current collection period, and the risk prediction is used to predict the running risk in a preset time period in the future according to all or part of the operation data up to the present.

[0074] As an example, the execution subject can perform abnormality detection on the display device according to the operation data by using a pre-trained abnormality detection model, and perform risk prediction on the display device according to the operation data by using a pre-trained risk prediction model.

[0075] In this implementation, the execution subject can perform the step 203 by performing the following operation: processing the multi-modal display data by using multiple monitoring strategies corresponding to the multi-modal display data by using the display data processing module in the target processing node, thereby obtaining the second monitoring result.

[0076] In this implementation, the target processing node for processing the operation data and the multi-modal display data is determined based on the load balancing of the multiple processing nodes in the data processing center, which helps to improve the data processing efficiency in a large-scale data scenario, ensures the orderly processing of large-scale data, and is further applicable to the monitoring scenario of multiple display devices.

[0077] In some optional implementations of the embodiment, the execution subject can execute the processing procedure of the running data to obtain the first monitoring result in the following manner:

[0078] First, the display device is subjected to abnormality detection according to the running data in combination with the preset detection rule and the abnormality detection model to obtain an abnormality detection result.

[0079] For example, the display device is subjected to abnormality detection according to the running data in combination with the preset detection rule to obtain a first abnormality detection result; the display device is subjected to abnormality detection according to the running data by using a pre-trained abnormality detection model to obtain a second abnormality detection result; and the abnormality detection result is obtained in combination with the first abnormality detection result and the second abnormality detection result. The combination of the first abnormality detection result and the second abnormality detection result can be, for example, summation, weighted summation, etc. The preset detection rule can be obtained based on the experience of the operation and maintenance personnel of the display device. The abnormality detection model can be, for example, a convolutional neural network or a recurrent neural network.

[0080] In some implementations, the abnormality detection model can be an ensemble learning model. In the ensemble learning model, multiple machine learning models are combined to work together to optimize the algorithm. These models can be multiple base learners trained on the same data set, which make predictions independently and then combine the predictions through a certain strategy (such as voting, averaging, etc.) to form a final prediction result, i.e., the second abnormality detection result.

[0081] Then, the display device is subjected to risk prediction according to the running data by using multiple risk prediction models to obtain a risk prediction result.

[0082] For example, for multiple pre-trained risk prediction models, the running data of a fixed time period before the decision time point is input, parallel integration prediction is performed, and a multi-classification result about risk prediction is output, which predicts whether there is a risk and the risk type in a fixed time period after the decision time point. The models all use a sliding training manner to ensure that new abnormal patterns can be learned. The sliding training means that the model is trained by gradually or continuously introducing new data.

[0083] Finally, the first monitoring result is obtained in combination with the abnormality detection result and the risk prediction result.

[0084] In the implementation, a specific determination manner of the first monitoring result is provided, which improves the accuracy and comprehensiveness of the first monitoring result based on abnormality detection and risk prediction.

[0085] In some optional implementations of the embodiment, the multi-modal presentation data includes sound data. In this implementation, the execution subject can execute the processing procedure of the sound data in the following manner:

[0086] Sound continuity detection: first, perform short-time Fourier transform on the sound data to determine the energy distribution information of the sound data in the time domain and the frequency domain; then, determine the sound continuity of the sound data according to the energy distribution information.

[0087] Specifically, the sound data is preprocessed, such as denoising and pre-emphasis, to improve the signal-to-noise ratio and highlight high-frequency components; the sound data is subjected to short-time Fourier transform to determine the energy distribution information of the sound data in the time domain and the frequency domain, and a time-frequency graph is obtained, which can directly show the frequency characteristics of the sound signal changing with time; features related to sound continuity are extracted from the time-frequency graph. These features may include spectral discontinuity, frequency component mutation, and amplitude sharp change. According to the extracted features, corresponding threshold values or rules are set to determine the sound continuity of the sound data. For example, whether there are obvious faults or mutations in the frequency spectrum, or whether the amplitude has undergone sharp changes in a short period of time, can be detected to determine whether there are stutters in the sound data, thereby determining the sound continuity.

[0088] Sound saturation detection: first, determine the Mel-frequency cepstral coefficients of the sound data; then, determine the sound saturation of the sound data according to the Mel-frequency cepstral coefficients.

[0089] Specifically, for a plurality of audio slices in time sequence, the MFCCs (Mel-Frequency Cepstral Coefficients) of the audio slices are calculated; and the mean and variance of the plurality of MFCCs are calculated to determine whether the sound saturation of the sound data is abnormal.

[0090] In this implementation, a specific monitoring strategy for sound data is provided, which improves the accuracy of the monitoring result of the sound data.

[0091] In some optional implementations of the embodiment, the multi-modal presentation data includes image data. In this implementation, the execution subject can execute the processing procedure of the image data in the following manner:

[0092] First, the perceptual hashing algorithm is used to compare the image data with the material images in the preset material library; then, in response to the comparison result indicating that the image data is beyond the image coverage range of the preset material library, an image sensitivity detection interface is called to determine the compliance of the image data.

[0093] The comparison process between the image data and the material images in the preset material library is as follows: first, the image data and the material images in the preset material library are adjusted to a fixed size (usually a smaller size, such as 32x32 or 64x64 pixels) and converted into a gray image. This is to simplify the calculation and reduce the generation time of the hash value. Then, a discrete cosine transform (DCT) is applied to the gray image. DCT is a technique commonly used in image compression, which can concentrate the energy of the image in a few coefficients. Then, a fixed size area (such as 8x8) is taken from the DCT transformed image and quantized to a binary value, i.e. a hash value. This is usually achieved by setting a threshold, for example, setting the coefficients greater than the threshold to 1 and the coefficients less than or equal to the threshold to 0. Finally, the hash value of the image data is compared with the hash value of the material images in the preset material library. The Hamming distance is usually used to measure the difference between two hash values.

[0094] In response to the hash values of the image data and the material images in the preset material library being different or having a large difference, it is determined that the image data is beyond the image coverage of the preset material library in response to the comparison result, i.e. the preset material library does not include the image data. Otherwise, it is determined that the preset material library includes the image data.

[0095] When the comparison result indicates that the preset material library does not include the image data, the image sensitive detection interface is continued to be called to determine the compliance of the image data. The image sensitive detection interface is used to detect whether the preset sensitive data is included in the image data.

[0096] In the implementation mode, a specific monitoring strategy for image data is provided, and the accuracy of the monitoring result of the image data is improved.

[0097] In some optional implementation modes of the embodiment, the multi-modal display data includes text data. In the implementation mode, the above-mentioned processing process of the text data can be performed by the following manner:

[0098] First, the text represented by the sound data and the text in the image data are recognized to obtain sound text data and image text data.

[0099] As an example, the above-mentioned execution subject can recognize the sound data by using the ASR (Automatic Speech Recognition) technology to obtain the sound text data, and recognize the text in the image data by using the OCR (Optical Character Recognition) technology to obtain the image text.

[0100] Then, the semantic analysis interface is called to determine whether sensitive data is included in the sound text data and the image text data, respectively.

[0101] The semantic analysis interface is used to perform semantic analysis on the text data to determine whether sensitive data is included therein.

[0102] Finally, in terms of timing, it is determined whether the subtitle text data in the sound text data and the image text data are synchronized.

[0103] In this implementation, the subtitle text data in the sound text data and the image text data are cooperatively detected to determine whether the sound text data and the image text data are synchronized. The subtitle text data represents subtitles in an image.

[0104] In this implementation, a specific monitoring strategy for text data is provided, which improves the accuracy of the monitoring result for the text data.

[0105] In some optional implementations of this embodiment, the multi-modal display data includes video data. In this implementation, the above-mentioned executing body can perform the processing process of the video data in the following manner:

[0106] First, based on a preset time interval, key frame images are collected from the video data to obtain a key frame sequence.

[0107] The preset time interval can be specifically set according to actual conditions, which is not limited herein.

[0108] Then, the similarity between adjacent key frame images in the key frame sequence is determined by using a perceptual hashing algorithm.

[0109] In this implementation, the above-mentioned executing body can refer to the processing process of the above-mentioned perceptual hashing algorithm to determine the hash value of each key frame image in the key frame sequence, and further determine the distance between the hash values of adjacent key frame images to determine the similarity between adjacent key frame images.

[0110] Finally, the video continuity of the video data is determined according to the similarity.

[0111] As an example, in response to the similarity between consecutive key frames continuously being higher than a preset threshold, it is determined that the video data has a lag and lacks continuity. The preset threshold can be specifically set according to actual conditions, which is not limited herein.

[0112] In this implementation, a specific monitoring strategy for video data is provided, which improves the accuracy of the monitoring result for the video data.

[0113] In some optional implementations of the embodiment, the execution subject can execute the step 201 by obtaining the running data and the multi-modal presentation data from the message queue.

[0114] In the implementation, the execution subject can execute the abnormal processing task corresponding to the abnormal situation by: firstly, transmitting the abnormal processing task to the message queue; and then, executing the abnormal processing task in the message queue by using the presentation device.

[0115] In the implementation, the message queue has the capability of bidirectional communication as the message hub of the execution subject (the server). The message queue is responsible for caching the running data and the multi-modal presentation data transmitted by the monitoring client, distributing and processing the request by using the load balancing mechanism, reducing the load pressure of the server, and caching the abnormal processing task issued by the server, and guaranteeing the order and real-time performance of the abnormal processing task by using the FIFO (First Input First Output) and the priority mechanism.

[0116] In the implementation, the bidirectional communication based on the message queue improves the information processing efficiency and the timeliness of the monitoring process.

[0117] With reference to Figure 4 , another embodiment of the monitoring method of the presentation device according to the present disclosure is shown in a schematic flow 400. In the flow 400, the following steps are included:

[0118] Step 401: obtaining, from the message queue, the running data and the multi-modal presentation data of the presentation device in the data presentation process.

[0119] Step 402: performing abnormal detection on the presentation device according to the running data by combining the preset detection rule and the abnormal detection model, to obtain an abnormal detection result.

[0120] Step 403: performing risk prediction on the presentation device according to the running data by using a plurality of risk prediction models, to obtain a risk prediction result.

[0121] Step 404: combining the abnormal detection result and the risk prediction result to obtain a first monitoring result.

[0122] Step 405: processing the multi-modal presentation data by using a plurality of monitoring strategies corresponding to the multi-modal presentation data one by one, to obtain a second monitoring result.

[0123] Step 406: in response to the first monitoring result and / or the second monitoring result indicating that there are a plurality of abnormal situations in the data presentation process, determining a plurality of abnormal processing strategies corresponding to the plurality of abnormal situations one by one.

[0124] Step 407: Using the strategy orchestration engine, orchestrate multiple exception handling strategies and determine exception handling tasks.

[0125] Step 408: In response to the failure of the policy orchestration engine to orchestrate multiple exception handling policies, an orchestration task for multiple exception handling policies is sent to the target terminal.

[0126] Step 409: Obtain the exception handling task based on the target terminal.

[0127] Step 410: Transmit the exception handling task to the message queue, and execute the exception handling task in the message queue through the display device.

[0128] It should be noted that the information processing procedures corresponding to steps 402-404 and the information processing procedure corresponding to step 405 can be executed in parallel or sequentially, and no limitation is made here.

[0129] The monitoring method for the display device in this embodiment, in process 400, compared to process 200, specifically describes the bidirectional communication process based on the message queue, the process for determining the monitoring results, and the process for determining the exception handling task. This reduces the intensity of manual monitoring while improving the security and reliability of the display process.

[0130] Continue to refer to Figure 5 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a monitoring system for a display device, which is similar to... Figure 2 Corresponding to the method embodiments shown, the system can be specifically applied to various electronic devices.

[0131] like Figure 5 As shown, the monitoring system 500 for the display device includes: a monitoring client 501, used to collect the operation data and multimodal display data of the display device during the data display process; a data processing center 502, used to monitor the operation status of the display device based on the operation data and obtain a first monitoring result; and to process the multimodal display data using multiple monitoring strategies that correspond one-to-one with the multimodal display data to obtain a second monitoring result; an anomaly handling center 503, used to respond to the indication of an anomaly in the data display process from the first monitoring result and / or the second monitoring result, and to determine the anomaly handling task corresponding to the anomaly; the monitoring client 501 is also used to control the display device to execute the anomaly handling task.

[0132] In some optional implementations of the present embodiment, the exception handling center 503 comprises an intelligent arrangement module, and the intelligent arrangement module is configured to: in response to the first monitoring result and / or the second monitoring result indicating that the data presentation process has multiple abnormal situations, determine multiple abnormal situation handling strategies corresponding to the multiple abnormal situations; and arrange the multiple abnormal situation handling strategies by using a strategy arrangement engine to determine the exception handling task.

[0133] In some optional implementations of the present embodiment, the exception handling center 503 comprises a manual arrangement module, and the manual arrangement module is configured to: in response to the strategy arrangement engine failing to arrange the multiple abnormal situation handling strategies, send an arrangement task of the multiple abnormal situation handling strategies to a target terminal; and based on the target terminal, obtain the exception handling task.

[0134] In some optional implementations of the present embodiment, the data processing center 502 is further configured to: determine a target processing node from the data processing center according to load data of processing nodes in the data processing center, wherein the target processing node comprises a running data processing module and a presentation data processing module, and the running data processing module is configured to: perform abnormality detection and risk prediction on the presentation device according to the running data to obtain the first monitoring result; and the presentation data processing module is configured to: process the multi-modal presentation data by using multiple monitoring strategies corresponding to the multi-modal presentation data to obtain the second monitoring result.

[0135] In some optional implementations of the present embodiment, the running data processing module is further configured to: perform abnormality detection on the presentation device according to the running data in combination with a preset detection rule and an abnormality detection model to obtain an abnormality detection result; perform risk prediction on the presentation device according to the running data by using multiple risk prediction models to obtain risk prediction results; and obtain the first monitoring result in combination with the abnormality detection result and the risk prediction results.

[0136] In some optional implementations of the present embodiment, the multi-modal presentation data comprises sound data, and the presentation data processing module comprises a sound processing submodule; and the sound processing submodule is configured to: perform short-time Fourier transform on the sound data to determine energy distribution information of the sound data in the time domain and the frequency domain; determine sound continuity of the sound data according to the energy distribution information; determine a mel-frequency cepstral coefficient of the sound data; and determine sound saturation of the sound data according to the mel-frequency cepstral coefficient.

[0137] In some optional implementations of the present embodiment, the multi-modal presentation data comprises image data, and the presentation data processing module comprises an image processing submodule; and the image processing submodule is configured to: compare the image data with material images in a preset material library by using a perceptual hashing algorithm; and in response to a comparison result indicating that the image data is beyond an image coverage range of the preset material library, call an image sensitivity detection interface to determine compliance of the image data.

[0138] In some optional implementations of the embodiment, the multi-modal presentation data includes text data, and the presentation data processing module includes a text processing submodule; and the text processing submodule is configured to: identify text in the sound data and text in the image data to obtain sound text data and image text data; call a semantic analysis interface to determine whether the sound text data and the image text data include sensitive data, respectively; and determine whether the sound text data, the image text data, and the subtitle text data are synchronized in time sequence.

[0139] In some optional implementations of the embodiment, the multi-modal presentation data includes video data, and the presentation data processing module includes a video processing submodule; and the video processing submodule is configured to: collect key frame images from the video data based on a preset time interval to obtain a key frame sequence; determine similarity between adjacent key frame images in the key frame sequence using a perceptual hashing algorithm; and determine video continuity of the video data according to the similarity.

[0140] In some optional implementations of the embodiment, the system further includes a message queue (not shown in the figure) configured to: transmit the running data and the multi-modal presentation data between the monitoring client and the data processing center; and transmit the abnormality processing task between the abnormality processing center and the monitoring client.

[0141] In the embodiment, a monitoring and broadcasting system of a presentation device is provided, which monitors and broadcasts in the device dimension and the content dimension based on the running data and the multi-modal presentation data of the presentation device, thereby improving the comprehensiveness of the monitoring and broadcasting process; a plurality of monitoring and broadcasting strategies corresponding to the multi-modal presentation data are used to process the multi-modal presentation data, thereby improving the pertinence and accuracy of the content dimension monitoring and broadcasting process; in response to an abnormal situation in the device dimension and / or the content dimension, an abnormality processing task corresponding to the abnormal situation is determined to process the abnormal situation in a targeted manner, thereby reducing the intensity of manual monitoring and broadcasting while improving the safety and reliability of the presentation process.

[0142] With continued reference to Figure 6 , a timing diagram inside the monitoring and broadcasting system of the presentation device is shown.

[0143] 1. Start the monitoring client and send an identity authentication request to the communication gateway.

[0144] 2. The communication gateway responds to the authentication result, and the monitoring client and the communication gateway establish an identity authentication data path.

[0145] 3. The monitoring client collects the running data and the multi-modal presentation data of the presentation device in the data presentation process, and sends the running data and the multi-modal presentation data to the communication gateway with authentication information.

[0146] 4. After the communication gateway completes the request identity authentication, the running data and the multi-modal display data are pushed into a bidirectional data message queue for caching.

[0147] 5. The processing node of the data processing center pulls the running data and the multi-modal display data in combination with its own load condition.

[0148] 6. In the processing node, the running data is processed by a running data processing module to obtain a first monitoring result, and the multi-modal display data is processed by a display data processing module to obtain a second monitoring result.

[0149] 7. The first monitoring result and the second monitoring result are sent to a dispatch center.

[0150] 8. In response to the first monitoring result and / or the second monitoring result indicating that an abnormal condition exists in the data display process, the abnormal processing center arranges a plurality of abnormal processing strategies corresponding to the plurality of abnormal conditions based on an intelligent arrangement module to obtain an abnormal processing task.

[0151] 9. In response to a failure of the strategy arrangement engine in arranging the plurality of abnormal processing strategies, the plurality of abnormal processing strategies are arranged by a manual arrangement module to obtain the abnormal processing task.

[0152] 10. The abnormal processing task is cached to a message queue.

[0153] 11. The message queue sends the sorted abnormal processing task to the communication gateway.

[0154] 12. The communication gateway feeds back the sorted abnormal processing task to a monitoring client;

[0155] 13. The monitoring client controls a monitoring terminal to execute the abnormal processing task.

[0156] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, comprising: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to implement the monitoring and broadcasting method of the display device described in any of the above embodiments when executed.

[0157] According to the embodiments of the present disclosure, the present disclosure further provides a readable storage medium storing computer instructions for enabling a computer to implement the monitoring and broadcasting method of the display device described in any of the above embodiments when executed.

[0158] The embodiments of the present disclosure provide a computer program product, which, when executed by a processor, can implement the monitoring and broadcasting method of the display device described in any of the above embodiments.

[0159] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0160] like Figure 7 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded from storage unit 708 into random access memory (RAM) 703. RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0161] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0162] The computing unit 701 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs various methods and processes described above, such as the monitoring method of a display device. For example, in some embodiments, the monitoring method of a display device can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded onto the RAM 703 and executed by the computing unit 701, one or more steps of the monitoring method of a display device described above can be performed. Alternatively, in other embodiments, the computing unit 701 can be configured to perform the monitoring method of a display device by any other suitable means, such as by means of firmware.

[0163] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0164] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a general purpose computer, special purpose computer, or other programmable processing or control devices to produce a machine, such that the program code, when executed by the processor or controller of the machine, causes the machine to perform the functions / acts specified in the flowcharts and / or block diagrams. The program code can execute entirely on a machine, partly on a machine, as a stand-alone software package, partly on a machine and partly on a remote machine or entirely on a remote machine or server.

[0165] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0166] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0167] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0168] The computer system can include clients and servers. This relationship can be. The servers are typically remote from the clients with the interactions typically happening over a communication network. This relationship can be created by the computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of large management difficulty and weak business scalability in traditional physical host and virtual private server (VPS, Virtual Private Server) services; it can also be a server of a distributed system, or a server combined with a blockchain.

[0169] According to the technical scheme of the embodiment of the present disclosure, a display device monitoring method and system are provided, which perform monitoring in the device dimension and the content dimension based on the running data of the display device and the multi-modal display data, thereby improving the comprehensiveness of the monitoring process; a plurality of monitoring strategies corresponding to the multi-modal display data are adopted to process the multi-modal display data, thereby improving the pertinence and accuracy of the content dimension monitoring process; in response to an abnormal situation existing in the device dimension and / or the content dimension, an abnormal processing task corresponding to the abnormal situation is determined to process the abnormal situation pertinently, thereby reducing the artificial monitoring intensity while improving the safety and reliability of the display process.

[0170] It should be understood that the various forms of flow shown above can be reordered, steps added or removed. For example, the steps described in the present disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical scheme provided by the present disclosure can be achieved, which is not limited herein.

[0171] The above detailed description does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A method for monitoring a display device, comprising: Acquire operational data and multimodal display data of the display device during the data display process; Based on the load data of the processing nodes in the data processing center, the target processing node is determined from the data processing center; The target processing node uses the running data processing module to perform anomaly detection and risk prediction on the display device based on the running data, and obtains the first monitoring result. The display data processing module in the target processing node uses multiple monitoring strategies corresponding to the multimodal display data to process the multimodal display data and obtain the second monitoring result. In response to the first monitoring result and / or the second monitoring result indicating that there are multiple abnormal situations in the data display process, determine multiple abnormal handling strategies corresponding to the multiple abnormal situations; The multiple exception handling strategies are orchestrated using a strategy orchestration engine, and exception handling tasks are determined and executed through the display device.

2. The method according to claim 1, wherein, Also includes: In response to the failure of the policy orchestration engine to orchestrate the multiple exception handling policies, the orchestration task of the multiple exception handling policies is sent to the target terminal. Based on the target terminal, obtain the exception handling task.

3. The method according to claim 1, wherein, The step of performing anomaly detection and risk prediction on the display device based on the operational data to obtain a first monitoring result includes: By combining preset detection rules and anomaly detection models, anomaly detection is performed on the display device based on the operational data to obtain anomaly detection results; The risk prediction results are obtained by using multiple risk prediction models to predict the risks of the display device based on the operational data. The first monitoring result is obtained by combining the anomaly detection results and risk prediction results.

4. The method according to claim 1, wherein, The multimodal display data includes audio data; as well as The process of processing the multimodal display data using multiple monitoring strategies that correspond one-to-one with the multimodal display data includes: Perform a short-time Fourier transform on the sound data to determine the energy distribution information of the sound data in the time domain and frequency domain; Based on the energy distribution information, determine the sound coherence of the sound data; Determine the Mel-frequency cepstral coefficients of the sound data; The sound saturation of the sound data is determined based on the Mel frequency cepstral coefficients.

5. The method according to claim 4, wherein, The multimodal display data includes image data; and The process of processing the multimodal display data using multiple monitoring strategies that correspond one-to-one with the multimodal display data includes: The image data is compared with images in a preset material library using a perceptual hash algorithm. In response to a comparison result indicating that the image data exceeds the image coverage range of the preset material library, the image sensitivity detection interface is invoked to determine the compliance of the image data.

6. The method according to claim 5, wherein, The multimodal display data includes text data; and The process of processing the multimodal display data using multiple monitoring strategies that correspond one-to-one with the multimodal display data includes: Identify the text represented by the sound data and the text in the image data to obtain sound text data and image text data; Call the semantic analysis interface to determine whether the audio text data and the image text data contain sensitive data, respectively; In terms of timing, it is determined whether the audio text data and the subtitle text data in the image text data are synchronized.

7. The method according to any one of claims 1, 3-6, wherein, The multimodal display data includes video data; and The process of processing the multimodal display data using multiple monitoring strategies that correspond one-to-one with the multimodal display data includes: Based on a preset time interval, key frame images are acquired from the video data to obtain a key frame sequence; The similarity between adjacent keyframe images in the keyframe sequence is determined using a perceptual hashing algorithm. The video coherence of the video data is determined based on the similarity.

8. The method according to claim 1, wherein, The acquisition of operational data and multimodal display data of the display device during the data display process includes: Retrieve the runtime data and the multimodal display data from the message queue; and The step of performing the exception handling task corresponding to the abnormal situation through the display device includes: The exception handling task is transmitted to the message queue; The exception handling task in the message queue is executed through the display device.

9. A monitoring system for a display device, comprising: The monitoring client is used to collect and display the operational data and multimodal display data of the devices during the data display process; A data processing center is used to determine a target processing node from the data processing center based on the load data of the processing nodes in the data processing center; through the running data processing module in the target processing node, anomaly detection and risk prediction are performed on the display device based on the running data to obtain a first monitoring result; through the display data processing module in the target processing node, multiple monitoring strategies corresponding one-to-one with the multimodal display data are used to process the multimodal display data to obtain a second monitoring result. The anomaly handling center includes an intelligent orchestration module, which is used to determine multiple anomaly handling strategies corresponding to the multiple anomalies in response to the first monitoring result and / or the second monitoring result indicating that there are multiple anomalies in the data display process. The multiple exception handling strategies are orchestrated through the strategy orchestration engine to determine the exception handling tasks. The monitoring client is also used to control the display device to perform the exception handling task.

10. The system according to claim 9, wherein, The anomaly handling center includes a manual orchestration module, and The manual arrangement module is used for: In response to the failure of the policy orchestration engine to orchestrate the multiple exception handling policies, the orchestration task of the multiple exception handling policies is sent to the target terminal. Based on the target terminal, obtain the exception handling task.

11. The system according to claim 9, wherein, The running data processing module is further used for: By combining preset detection rules and anomaly detection models, anomaly detection is performed on the display device based on the operational data to obtain anomaly detection results; and multiple risk prediction models are used to predict the risks of the display device based on the operational data to obtain risk prediction results. The first monitoring result is obtained by combining the anomaly detection results and risk prediction results.

12. The system according to claim 9, wherein, The multimodal display data includes audio data, and the display data processing module includes an audio processing submodule; and The sound processing submodule is used for: Perform a short-time Fourier transform on the sound data to determine the energy distribution information of the sound data in the time and frequency domains; determine the sound coherence of the sound data based on the energy distribution information; determine the Mel frequency cepstral coefficients of the sound data; and determine the sound saturation of the sound data based on the Mel frequency cepstral coefficients.

13. The system according to claim 12, wherein, The multimodal display data includes image data, and the display data processing module includes an image processing submodule; and The image processing submodule is used for: The image data is compared with images in a preset material library using a perceptual hash algorithm. In response to a comparison result indicating that the image data exceeds the image coverage of the preset material library, the image sensitivity detection interface is invoked to determine the compliance of the image data.

14. The system according to claim 13, wherein, The multimodal display data includes text data, and the display data processing module includes a text processing submodule; and The text processing submodule is used for: Identify the text represented by the sound data and the text in the image data to obtain sound text data and image text data; call the semantic analysis interface to determine whether the sound text data and the image text data contain sensitive data; and determine whether the subtitle text data in the sound text data and the image text data are synchronized in terms of timing.

15. The system according to any one of claims 9, 11-14, wherein, The multimodal display data includes video data, and the display data processing module includes a video processing submodule; and The video processing submodule is used for: Based on a preset time interval, keyframe images are acquired from the video data to obtain a keyframe sequence; a perceptual hash algorithm is used to determine the similarity between adjacent keyframe images in the keyframe sequence; and the video coherence of the video data is determined based on the similarity.

16. The system according to claim 9, wherein, Also includes: Message queues are used for: The operational data and the multimodal display data are transmitted between the monitoring client and the data processing center; The exception handling task is transmitted between the exception handling center and the monitoring client.

17. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

18. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

19. A computer program product comprising: A computer program that, when executed by a processor, implements the method according to any one of claims 1-8.

Images