Display device monitoring method and system, and computer program product

Abstract

The invention provides a broadcast monitoring method and system of display equipment, electronic equipment, a storage medium and a computer program product, relates to the technical field of computers, in particular to the technical fields of large language models, image recognition, natural language processing, terminal broadcast monitoring and the like, and can be applied to broadcast monitoring scenes. According to the specific implementation scheme, the method comprises the steps of obtaining operation data and multi-modal display data of display equipment in a data display process; monitoring the operation state of the display equipment according to the operation data to obtain a first monitoring result; processing the multi-modal display data by adopting a plurality of monitoring strategies in one-to-one correspondence with the multi-modal display data to obtain a second monitoring result; and in response to the first monitoring result and / or the second monitoring result indicating that an abnormal condition exists in the data display process, determining and executing an exception handling task corresponding to the abnormal condition through the display equipment. According to the invention, the artificial broadcast monitoring intensity is reduced, and the safety and reliability of the display process are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, specifically to technical fields such as large language models, image recognition, natural language processing, and terminal monitoring, and in particular to a monitoring method, system, electronic device, storage medium, and computer program product for a display device, which can be applied in monitoring scenarios. Background Art

[0002] With the continuous development of large language model technology, applications equipped with AIGC (Artificial Intelligence Generated Content) technology have been widely used in digital exhibition hall design. These applications can not only interpret the information, background and practical value of scientific and technological exhibits through the interactive form of natural language, but also answer the questions of visitors in a dialogue way, so that the audience can get a more personalized, intelligent and interactive exhibition experience, further enhancing the attractiveness and experience of digital exhibitions. As an important medium for information dissemination and interaction in science and technology exhibition halls, the accuracy and fluency of the display content of digital screens will directly affect the audience's experience and information dissemination effect. However, due to the large number of display equipment terminals in the exhibition hall and the certain degree of freedom of the display content, it is difficult to achieve full coverage monitoring by using manual patrols. Summary of the invention

[0003] The present disclosure provides a method and apparatus for monitoring a display device, an electronic device, a storage medium, and a computer program product.

[0004] According to a first aspect, a monitoring method for a display device is provided, comprising: obtaining operating data and multimodal display data of the display device during a data display process; monitoring the operating status of the display device according to the operating data to obtain a first monitoring result; adopting a plurality of monitoring strategies corresponding one-to-one to the multimodal display data to process the multimodal display data to obtain a second monitoring result; in response to the first monitoring result and / or the second monitoring result indicating that an abnormality exists in the data display process, determining and executing, through the display device, an exception handling task corresponding to the abnormality.

[0005] According to a second aspect, a monitoring system for a display device is provided, comprising: a monitoring client, used to collect operating data and multimodal display data of the display device during a data display process; a data processing center, used to monitor the operating status of the display device according to the operating data to obtain a first monitoring result; a plurality of monitoring strategies corresponding one-to-one to the multimodal display data are adopted to process the multimodal display data to obtain a second monitoring result; an exception handling center, used to determine an exception handling task corresponding to the abnormal situation in response to the first monitoring result and / or the second monitoring result indicating that an abnormal situation exists in the data display process; and the monitoring client, further used to control the display device to execute the exception handling task.

[0006] According to a third aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a method described in any implementation manner of the first aspect.

[0007] According to a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to execute the method described in any implementation of the first aspect.

[0008] According to a fifth aspect, a computer program product is provided, comprising: a computer program, which implements the method described in any implementation manner of the first aspect when executed by a processor.

[0009] According to the technology disclosed in the present invention, a monitoring method and system for display equipment are provided, which monitor the equipment dimension and content dimension based on the operation data and multimodal display data of the display equipment, thereby improving the comprehensiveness of the monitoring process; adopt multiple monitoring strategies that correspond one-to-one to the multimodal display data to process the multimodal display data, thereby improving the pertinence and accuracy of the content dimension monitoring process; in response to abnormal situations in the equipment dimension and / or content dimension, determine the abnormal situation The corresponding exception handling task is handled in a targeted manner, thereby reducing the intensity of manual monitoring while improving the safety and reliability of the display process.

[0010] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0012] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present disclosure may be applied;

[0013] Figure 2 is a flow chart of an embodiment of a monitoring method of a display device according to the present disclosure;

[0014] Figure 3 A schematic diagram of an application scenario of the monitoring method for a display device according to this embodiment;

[0015] Figure 4 is a flow chart of another embodiment of a monitoring method of a display device according to the present disclosure;

[0016] Figure 5 is a structural diagram of an embodiment of a monitoring system for a display device according to the present disclosure;

[0017] Figure 6 It is a time chart of the monitoring system of the display device according to the present disclosure.

[0018] Figure 7 It is a schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure. DETAILED DESCRIPTION

[0019] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0020] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0021] Figure 1 An exemplary architecture 100 is shown to which the monitoring method and system of a display device disclosed in the present invention can be applied.

[0022] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The communication connection between the terminal devices 101, 102, 103 constitutes a topological network, and the network 104 is used to provide a medium for the communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0023] Terminal devices 101, 102, 103 may be hardware devices or software that support network connection for data interaction and data processing. When terminal devices 101, 102, 103 are hardware, they may be various electronic devices that support network connection, information acquisition, interaction, display, processing and other functions, including but not limited to display devices, digital screens, smart phones, tablet computers, e-book readers, laptop portable computers and desktop computers, etc. When terminal devices 101, 102, 103 are software, they may be installed in the electronic devices listed above. They may be implemented as multiple software or software modules, for example, for providing distributed services, or they may be implemented as a single software or software module. No specific limitation is made here.

[0024] The server 105 may be a server that provides various services, for example, a background processing server that obtains the operation data and multimodal display data of the terminal devices 101, 102, 103 during the data display process and monitors the data display process. As an example, the server 105 may be a cloud server.

[0025] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, software or software modules used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0026] It should also be noted that the monitoring method of the display device provided in the embodiments of the present disclosure is generally executed by a server, but the possibility of being executed by a terminal device, or being executed by the server and the terminal device in cooperation with each other is not excluded. Accordingly, the various parts (such as various units) included in the monitoring system of the display device can be all set in the server, or all set in the terminal device, or can be set in the server and the terminal device respectively.

[0027] It should be understood that Figure 1 The number of terminal devices, networks and servers in the above description is only for illustration. Any number of terminal devices, networks and servers may be provided as required. When the electronic device on which the monitoring method of the display device is running does not need to perform data transmission with other electronic devices, the system architecture may only include the electronic device (e.g., terminal device or server) on which the monitoring method of the display device is running.

[0028] Please refer to Figure 2 , Figure 2 A flowchart of a method for monitoring broadcasting of a display device provided in an embodiment of the present disclosure. In process 200, the following steps are included:

[0029] Step 201: Acquire operation data and multimodal display data of a display device during data display.

[0030] In this embodiment, the execution subject of the monitoring method of the display device (for example, Figure 1 The server in the data display device can obtain the operation data and multimodal display data of the display device during the data display process remotely or locally through a wired network connection method or a wireless network connection method.

[0031] The display device is a media device that displays multimodal display data and supports multiple operating systems (including Windows, Linux and other operating systems with graphical interfaces), such as multiple digital screens in a digital exhibition hall. A monitoring client is deployed in the display device, and the monitoring client can collect the operating data and multimodal display data of the display device during the data display process. The above-mentioned execution entity can obtain the operating data and multimodal display data collected by the monitoring client. The display device with the monitoring client deployed locally is the monitoring terminal.

[0032] Operation data includes static data and dynamic data of the display device. Static data is generally device attribute data, including but not limited to device ID (Identity document), GPU (Graphics Processing Unit) model, graphics card driver version, and device asset scrapping date. Dynamic data is generally process data of the display device during operation, including but not limited to device operation time, device temperature, device power consumption, GPU stream processor utilization, GPU video memory utilization, and device operation process.

[0033] Multimodal presentation data may be presentation data in various modes, including but not limited to presentation data in modes such as text, sound, image, and video.

[0034] In this implementation, in each data collection cycle, all operation data and all multimodal display data in the cycle may be collected, or part of the operation data and part of the multimodal display data in the cycle may be collected. For example, at every preset time interval, a collection operation of operation data and multimodal display data is performed to obtain part of the operation data and part of the multimodal display data in one collection cycle.

[0035] Different preset time intervals may be set for different operating data and display data of different modes, so as to specifically adapt the preset time intervals to different data.

[0036] In some implementations, the monitoring server corresponding to the above-mentioned execution entity includes a communication gateway.

[0037] The communication gateway is the hub between the monitoring client and the monitoring server. It has the ability of two-way communication authentication and is responsible for processing and distributing data transmission requests from the monitoring client to ensure the security and integrity of the data flow.

[0038] Specifically, the deployment and authentication process of the monitoring client is as follows: First, the display device starts the monitoring client and keeps the monitoring client running through the daemon process; then, the monitoring client generates a unique device identification code, sends an identity signature authentication request to the communication gateway, and applies for signature authentication. Then, the monitoring gateway receives the identity signature authentication request, and after authentication, stores the authentication mapping relationship between the monitoring client and the display device. Then, the communication gateway sends an authentication request (including signature authentication information and global configuration parameters of the monitoring process) to the monitoring client. Finally, the monitoring client receives the authentication response from the communication gateway and successfully establishes a two-way data transmission channel with the communication gateway.

[0039] Step 202: monitor the operating status of the display device according to the operating data to obtain a first monitoring result.

[0040] In this embodiment, the execution subject may monitor the operation status of the display device according to the operation data to obtain a first monitoring result.

[0041] As an example, the execution subject may monitor the operating status of the display device based on the prior strategy formed by the experience of the equipment maintenance personnel, determine whether the operating status of the display device is abnormal, and obtain the first monitoring result. The prior strategy is, for example, a judgment rule for various abnormal situations during the operation of the equipment.

[0042] As another example, the execution subject may input the operation data into a pre-trained operation status monitoring model to determine whether the operation status of the display device is abnormal, and obtain a first monitoring result. The operation status detection model represents the corresponding relationship between the operation data and the first monitoring result, such as a recurrent neural network, a residual network, or other models.

[0043] Step 203: adopt multiple monitoring strategies corresponding to the multimodal display data one by one, process the multimodal display data, and obtain a second monitoring result.

[0044] In this embodiment, the execution subject may adopt a plurality of monitoring strategies corresponding to the multimodal display data one by one, process the multimodal display data, and obtain the second monitoring result.

[0045] Specifically, for the display data of each modality in the multimodal display data, the display data of the modality is processed through the monitoring strategy corresponding to the display data of the modality to obtain the sub-monitoring result corresponding to the display data of the modality; and the multiple sub-monitoring results corresponding to the multimodal display data are combined to obtain the second monitoring result.

[0046] As an example, the monitoring strategy set includes multiple monitoring strategies, including but not limited to a monitoring strategy for determining whether the data is smooth, a monitoring strategy for determining whether the data includes sensitive content, a monitoring strategy for determining whether the data of each modality is synchronized, and a monitoring strategy for determining whether the data is logical. For each modal display data in the multimodal display data, a target monitoring strategy corresponding to the display data of the modality is determined from the monitoring strategy set, and the display data of the modality is processed by the target monitoring strategy to obtain a sub-monitoring result corresponding to the display data of the modality.

[0047] As another example, multiple monitoring models corresponding to multimodal display data are used to process the multimodal display data to obtain a second monitoring result. For each modal display data in the multimodal display data, the display data of the modality is processed by the monitoring model corresponding to the display data of the modality to obtain a sub-monitoring result corresponding to the display data of the modality. The model corresponding to the display data of the modality is used to execute the monitoring strategy corresponding to the display data of the modality. Then, the second monitoring result is obtained by combining the multiple sub-monitoring results corresponding to the multimodal display data.

[0048] Taking the monitoring strategy for determining whether data includes sensitive content as an example, the above-mentioned execution subject can use the training sample set corresponding to the detection strategy to train the monitoring model. Specifically, first, a training sample set is obtained, wherein the training samples in the training sample set include sample data, a judgment label representing whether the sample data includes sensitive data, and a sensitive data label representing the sensitive data when sensitive data is included. Then, a machine learning algorithm is used, with the sample data as the output, and the judgment label and sensitive data label corresponding to the sample data as the expected output, to train the detection model.

[0049] It should be noted that the information processing process corresponding to the above step 202 and the information processing process corresponding to the above step 203 can be executed in parallel or sequentially, which is not limited here.

[0050] Step 204: in response to the first monitoring result and / or the second monitoring result indicating that an abnormality exists in the data display process, determining and executing an abnormality handling task corresponding to the abnormality through the display device.

[0051] In this embodiment, the execution subject can determine and execute the exception handling task corresponding to the abnormal situation through the display device in response to the first monitoring result and / or the second monitoring result indicating that there is an abnormal situation in the data display process. For example, an exception handling center is set in the execution subject, and the exception handling center can determine the exception handling task corresponding to the abnormal situation.

[0052] As an example, the execution subject or the electronic device connected to the execution subject in communication is provided with corresponding relationship data representing the corresponding relationship between the abnormal situation and the abnormality handling task. In response to the first monitoring result and / or the second monitoring result indicating that there is an abnormal situation in the data display process, the abnormality handling task corresponding to the abnormal situation is determined from the corresponding relationship data, and the abnormality handling task corresponding to the abnormal situation is executed by the display device to resolve the abnormal situation.

[0053] As another example, the execution subject may determine the exception handling task corresponding to the exception situation based on the exception handling model. For example, the exception handling model is a deep learning network model or a large language model with exception handling capability.

[0054] In some optional implementations of this embodiment, an exception handling center is set in the execution subject, and the exception handling center can perform the above step 204 in the following manner:

[0055] The first step is to determine a plurality of exception handling strategies corresponding to the plurality of exception situations in response to the first monitoring result and / or the second monitoring result indicating that the data display process has a plurality of exception situations.

[0056] In this implementation, for each abnormal situation, a corresponding abnormality handling strategy is set. The abnormality handling strategy is, for example, an operation execution path that can solve the abnormal situation. The abnormal situation can be solved by executing each operation in the operation execution path in sequence.

[0057] As an example, for the abnormal situation that the demonstration program crashes, the exception handling strategy is to reset the system process -> start the demonstration program -> initialize the demonstration program; for the abnormal situation that the demonstration program is out of synchronization, the exception handling strategy is to refresh and reset the demonstration program; for the abnormal situation that the audio channel of the demonstration program is missing, the exception handling strategy is hardware self-check -> sound card driver self-check / install / update -> start the sound card driver -> audio adaptive adjustment.

[0058] For multiple abnormal situations in the first monitoring result and / or the second monitoring result, multiple abnormality handling strategies corresponding to the multiple abnormal situations are determined.

[0059] The second step is to orchestrate multiple exception handling strategies and determine the exception handling tasks through the policy orchestration engine.

[0060] By defining the dependencies and execution order between policies, the policy orchestration engine can automatically schedule and execute a series of exception handling policies, thereby improving work efficiency, reducing costs, and improving service quality.

[0061] In this implementation, a plurality of exception handling strategies are orchestrated through a policy orchestration engine, for example, a plurality of operation paths corresponding to the plurality of exception handling strategies are organized, coordinated and integrated according to certain logic, rules or processes, thereby forming a complete, orchestrated operation path, i.e., an exception handling task.

[0062] In this implementation, the exception handling center includes an intelligent orchestration module, and the first step and the second step can be executed by the intelligent orchestration module.

[0063] In this implementation, a specific method for determining an exception handling task is provided, which can cope with complex scenarios in which the data display process includes multiple exception situations at the same time, thereby improving the efficiency and capability of handling exception situations.

[0064] In some optional implementations of this embodiment, the execution subject may perform step 204 in the following manner:

[0065] The third step is to send the orchestration tasks of the multiple exception handling policies to the target terminal in response to the failure of the policy orchestration engine to orchestrate the multiple exception handling policies. The fourth step is to obtain the exception handling tasks based on the target terminal.

[0066] In the case where automatic orchestration is not supported or an exception occurs during the orchestration process and the orchestration fails, the manual orchestration module will be triggered to send the orchestration tasks of multiple exception handling strategies to the target terminal, and the on-duty operation and maintenance personnel will be notified to intervene and uniformly orchestrate multiple exception handling strategies to ensure that when the system's self-healing capability is insufficient, the potential impact can be reduced through manual intervention.

[0067] In this implementation, based on the manual arrangement method, the orderly handling of abnormal situations is guaranteed, and the safety and reliability of the monitoring process are further improved.

[0068] Continue to see Figure 3 , Figure 3 It is a schematic diagram of an application scenario 300 of the monitoring method for display devices according to the present embodiment. The server 301 first obtains the operation data and multimodal display data of each of the multiple display devices 302 during the data display process. For each display device, the operation status of the display device is monitored according to the operation data of the display device to obtain a first monitoring result; multiple monitoring strategies corresponding to the multimodal display data are adopted to process the multimodal display data corresponding to the operating device to obtain a second monitoring result; finally, in response to the first monitoring result and / or the second monitoring result indicating that there is an abnormality in the data display process, the abnormality handling task corresponding to the abnormality is determined and executed through the display device.

[0069] In the present embodiment, a monitoring method for a display device is provided, which performs monitoring in the device dimension and the content dimension based on the operation data and multimodal display data of the display device, thereby improving the comprehensiveness of the monitoring process; adopts multiple monitoring strategies that correspond one-to-one to the multimodal display data to process the multimodal display data, thereby improving the pertinence and accuracy of the content dimension monitoring process; in response to abnormal situations in the device dimension and / or content dimension, determines the exception handling task corresponding to the abnormal situation, so as to handle the abnormal situation in a targeted manner, thereby reducing the intensity of manual monitoring while improving the safety and reliability of the display process.

[0070] In some optional implementations of this embodiment, the above-mentioned execution subject may also perform the following operations: determining a target processing node from the data processing center according to load data of the processing nodes in the data processing center.

[0071] The data processing center includes a plurality of processing nodes, each of which includes an operation data processing module and a display data processing module. As an example, the execution subject may determine a target processing node from the data processing center based on a load balancing strategy and according to load data of the processing nodes in the data processing center.

[0072] In this implementation, the execution subject may perform step 202 in the following manner: by using the operation data processing module in the target processing node, anomaly detection and risk prediction are performed on the display device according to the operation data to obtain a first monitoring result.

[0073] Among them, anomaly detection is used to detect whether an abnormality occurs during the operation of the display equipment during the current collection cycle, and risk prediction is used to predict the operation risk in a preset time period in the future based on all or part of the operation data up to the present.

[0074] As an example, the above-mentioned execution entity can perform anomaly detection on the display equipment according to the operation data through a pre-trained anomaly detection model, and perform risk prediction on the display equipment according to the operation data through a pre-trained risk prediction model.

[0075] In this implementation, the execution subject may perform step 203 as follows: by using a display data processing module in a target processing node, a plurality of monitoring strategies corresponding to multimodal display data are adopted to process the multimodal display data to obtain a second monitoring result.

[0076] In this implementation, based on the load balancing of multiple processing nodes in the data processing center, the target processing node for processing the operation data and multimodal display data is determined, which helps to improve the data processing efficiency in large-scale data scenarios, ensure the orderly processing of large-scale data, and is further suitable for monitoring scenarios for multiple display devices.

[0077] In some optional implementations of this embodiment, the execution subject may execute the processing of the operation data in the following manner to obtain the first monitoring result:

[0078] Firstly, the preset detection rules and anomaly detection model are combined to perform anomaly detection on the display equipment according to the operation data to obtain the anomaly detection results.

[0079] As an example, based on the operation data, the display device is detected for anomalies using preset detection rules to obtain a first anomaly detection result; based on the operation data, the display device is detected for anomalies using a pre-trained anomaly detection model to obtain a second anomaly detection result; the first anomaly detection result and the second anomaly detection result are combined to obtain an anomaly detection result. The first anomaly detection result and the second anomaly detection result are combined, for example, by summing, weighted summing, etc. The preset detection rules can be obtained based on the experience of the operation and maintenance personnel of the display device. The anomaly detection model is, for example, a convolutional neural network or a recurrent neural network.

[0080] In some implementations, the anomaly detection model may be an ensemble learning model. In an ensemble learning model, multiple machine learning models are combined to work together to achieve the purpose of optimizing the algorithm. These models may be multiple base learners trained on the same data set, each of which makes predictions independently, and then these prediction results are combined through a certain strategy (such as voting, averaging, etc.) to form a final prediction result, i.e., the second anomaly detection result.

[0081] Then, multiple risk prediction models are used to perform risk prediction on the display equipment according to the operating data to obtain risk prediction results.

[0082] As an example, for multiple pre-trained risk prediction models, the operating data of a fixed period before the decision time point is input, and the prediction is integrated in parallel to output multi-classification results on risk prediction, which predicts whether there is a risk and the type of risk in a fixed period after the decision time point. The models all use a sliding training method to ensure that new abnormal patterns can be learned. Sliding training refers to training the model by gradually or continuously introducing new data.

[0083] Finally, the first monitoring result is obtained by combining the anomaly detection results and risk prediction results.

[0084] In this implementation, a specific method for determining a first monitoring result is provided, which improves the accuracy and comprehensiveness of the first monitoring result based on anomaly detection and risk prediction.

[0085] In some optional implementations of this embodiment, the multimodal presentation data includes sound data. In this implementation, the execution subject may perform the processing of the sound data in the following manner:

[0086] Sound continuity detection: First, perform short-time Fourier transform on the sound data to determine the energy distribution information of the sound data in the time domain and frequency domain; then, determine the sound continuity of the sound data based on the energy distribution information.

[0087] Specifically, the sound data is preprocessed by denoising, pre-emphasis and other preprocessing operations to improve the signal-to-noise ratio and highlight the high-frequency components; the sound data is short-time Fourier transformed to determine the energy distribution information of the sound data in the time domain and frequency domain, and obtain its time-frequency diagram, which can intuitively show the frequency characteristics of the sound signal changing over time; and features related to sound coherence are extracted from the time-frequency diagram. These features may include discontinuity of the spectrum, sudden changes in frequency components, and sharp changes in amplitude. According to the extracted features, corresponding thresholds or rules are set to determine the sound coherence of the sound data. For example, it is possible to detect whether there are obvious faults or sudden changes in the spectrum, or whether the amplitude has changed sharply in a short period of time, to determine whether there is a freeze in the sound data, thereby determining the sound coherence.

[0088] Sound saturation detection: First, determine the Mel-frequency cepstral coefficients of the sound data; then, determine the sound saturation of the sound data based on the Mel-frequency cepstral coefficients.

[0089] Specifically, for multiple audio slices that are continuous in time sequence, MFCCs (Mel-Frequency Cepstral Coefficients) of the audio slices are calculated; and then the mean and variance of the multiple MFCCs are calculated to determine whether the sound saturation of the sound data is abnormal.

[0090] In this implementation, a specific monitoring strategy for sound data is provided, which improves the accuracy of the monitoring results for sound data.

[0091] In some optional implementations of this embodiment, the multimodal display data includes image data. In this implementation, the execution subject may perform the processing of the image data in the following manner:

[0092] First, a perceptual hash algorithm is used to compare the image data with the material images in the preset material library; then, in response to the comparison result indicating that the image data exceeds the image coverage of the preset material library, the image sensitive detection interface is called to determine the compliance of the image data.

[0093] The comparison process between the image data and the material images in the preset material library is as follows: First, the image data and the material images in the preset material library are adjusted to a fixed size (usually a smaller size, such as 32x32 or 64x64 pixels) and converted into grayscale images. This is to simplify the calculation and reduce the generation time of the hash value. Then, the discrete cosine transform (DCT) is applied to the grayscale image. DCT is a technology commonly used in image compression, which can concentrate the energy of the image on a few coefficients. Then, from the DCT-transformed image, a fixed-size area (such as 8x8) is taken and quantized into a binary value, namely a hash value. This is usually achieved by setting a threshold, such as setting the coefficient greater than the threshold to 1 and the coefficient less than or equal to the threshold to 0. Finally, the hash value of the image data is compared with the hash value of the material image in the preset material library. The Hamming distance is usually used to measure the difference between two hash values.

[0094] In response to the image data and each material image in the preset material library having different or large differences in hash values, it is determined that the image data exceeds the image coverage range of the preset material library in response to the comparison result, that is, the preset material library does not include the image data. Otherwise, it is determined that the preset material library includes the image data.

[0095] When the comparison result indicates that the preset material library does not include the image data, the image sensitive data detection interface is called to determine the compliance of the image data. The image sensitive data detection interface is used to detect whether the image data includes the preset sensitive data.

[0096] In this implementation, a specific monitoring strategy for image data is provided, which improves the accuracy of the monitoring results for image data.

[0097] In some optional implementations of this embodiment, the multimodal display data includes text data. In this implementation, the execution subject may perform the processing of the text data in the following manner:

[0098] First, the text represented by the sound data and the text in the image data are recognized to obtain the sound text data and the image text data.

[0099] As an example, the above-mentioned execution subject can use ASR (Automatic Speech Recognition) technology to recognize sound data and obtain sound text data; use OCR (Optical Character Recognition) technology to recognize text in image data and obtain image text.

[0100] Then, the semantic analysis interface is called to determine whether the sound text data and the image text data include sensitive data.

[0101] The semantic analysis interface is used to perform semantic analysis on text data to determine whether it contains sensitive data.

[0102] Finally, in terms of timing, it is determined whether the audio text data and the subtitle text data in the image text data are synchronized.

[0103] In this implementation, the audio text data and the subtitle text data in the image text data are cooperatively detected to determine whether the audio text data and the image text data are synchronized. The subtitle text data represents the subtitles in the image.

[0104] In this implementation, a specific monitoring strategy for text data is provided, which improves the accuracy of the monitoring results for text data.

[0105] In some optional implementations of this embodiment, the multimodal presentation data includes video data. In this implementation, the execution subject may perform the processing of the video data in the following manner:

[0106] Firstly, based on a preset time interval, key frame images are collected from the video data to obtain a key frame sequence.

[0107] The preset time interval can be set according to actual conditions and is not limited here.

[0108] Then, the perceptual hashing algorithm is used to determine the similarity between adjacent key frame images in the key frame sequence.

[0109] In this implementation, the above-mentioned execution entity can refer to the processing process of the above-mentioned perceptual hash algorithm to determine the hash value of each key frame image in the key frame sequence, and then determine the distance between the hash values ​​of adjacent key frame images to determine the similarity between adjacent key frame images.

[0110] Finally, the video coherence of the video data is determined based on the similarity.

[0111] As an example, in response to the similarity between consecutive key frames being continuously higher than a preset threshold, it is determined that the video data is stuck and the video lacks coherence. The preset threshold can be specifically set according to actual conditions and is not limited here.

[0112] In this implementation, a specific monitoring strategy for video data is provided, which improves the accuracy of monitoring results for video data.

[0113] In some optional implementations of this embodiment, the execution subject may perform step 201 in the following manner: obtaining operation data and multimodal display data from a message queue.

[0114] In this implementation, the above-mentioned execution subject can execute the exception handling task corresponding to the abnormal situation in the following manner: first, the exception handling task is transmitted to the message queue; then, the exception handling task in the message queue is executed through the display device.

[0115] In this implementation, the message queue serves as the message hub between the communication gateway and the above-mentioned execution subject (server), and has the ability of two-way communication. The message queue is responsible for caching the operation data and multi-modal display data transmitted by the monitoring client, using the load balancing mechanism to distribute processing requests and reduce the load pressure on the server, and is also responsible for caching the exception handling tasks issued by the server, using the FIFO (First Input First Output) and priority mechanism to ensure the sequential and real-time push of the exception handling tasks.

[0116] In this implementation, the two-way communication based on the message queue improves the information processing efficiency and timeliness of the monitoring process.

[0117] Continue to refer Figure 4 , shows a schematic process 400 of another embodiment of a monitoring method of a display device according to the present disclosure. In the process 400, the following steps are included:

[0118] Step 401: Acquire the operation data and multimodal display data of the display device during the data display process from the message queue.

[0119] Step 402 , combining the preset detection rules and the anomaly detection model, performing anomaly detection on the display device according to the operation data, and obtaining anomaly detection results.

[0120] Step 403: perform risk prediction on the display equipment according to the operation data using multiple risk prediction models to obtain risk prediction results.

[0121] Step 404: Combine the anomaly detection result and the risk prediction result to obtain a first monitoring result.

[0122] Step 405 , using a plurality of monitoring strategies corresponding to the multimodal display data one by one, processing the multimodal display data to obtain a second monitoring result.

[0123] Step 406: In response to the first monitoring result and / or the second monitoring result indicating that there are multiple abnormal situations in the data display process, multiple abnormal handling strategies corresponding to the multiple abnormal situations are determined.

[0124] Step 407: orchestrate multiple exception handling strategies through a policy orchestration engine to determine the exception handling task.

[0125] Step 408: In response to the policy orchestration engine failing to orchestrate the multiple exception handling policies, an orchestration task of the multiple exception handling policies is sent to the target terminal.

[0126] Step 409: Acquire an exception handling task based on the target terminal.

[0127] Step 410, transferring the exception handling task to the message queue, and executing the exception handling task in the message queue through the display device.

[0128] It should be noted that the information processing process corresponding to the above steps 402-404 and the information processing process corresponding to the above step 405 can be executed in parallel or sequentially, which is not limited here.

[0129] Compared with process 200, process 400 of the monitoring method of the display device in this embodiment specifically describes the two-way communication process based on the message queue, the process of determining the monitoring results, and the process of determining the exception handling tasks, which reduces the intensity of manual monitoring while improving the safety and reliability of the display process.

[0130] Continue to refer Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a monitoring system for display equipment, and the system embodiment is Figure 2 Corresponding to the method embodiment shown, the system can be specifically applied to various electronic devices.

[0131] like Figure 5 As shown, the monitoring system 500 of the display device includes: a monitoring client 501, which is used to collect the operating data and multimodal display data of the display device during the data display process; a data processing center 502, which is used to monitor the operating status of the display device according to the operating data to obtain a first monitoring result; adopt a plurality of monitoring strategies corresponding to the multimodal display data one by one, process the multimodal display data, and obtain a second monitoring result; an exception handling center 503, which is used to respond to the first monitoring result and / or the second monitoring result indicating that there is an abnormality in the data display process, and determine the exception handling task corresponding to the abnormality; the monitoring client 501 is also used to control the display device to perform the exception handling task.

[0132] In some optional implementations of this embodiment, the exception handling center 503 includes an intelligent orchestration module, and an intelligent orchestration module for: in response to the first monitoring result and / or the second monitoring result indicating that there are multiple abnormal situations in the data display process, determining multiple exception handling strategies corresponding to the multiple abnormal situations; through a strategy orchestration engine, orchestrating multiple exception handling strategies and determining exception handling tasks.

[0133] In some optional implementations of this embodiment, the exception handling center 503 includes a manual orchestration module, and the manual orchestration module is used to: in response to the failure of the policy orchestration engine to orchestrate multiple exception handling policies, send an orchestration task of multiple exception handling policies to the target terminal; based on the target terminal, obtain the exception handling task.

[0134] In some optional implementations of the present embodiment, the data processing center 502 is further used to: determine a target processing node from the data processing center according to the load data of the processing nodes in the data processing center, wherein the target processing node includes an operation data processing module and a display data processing module, and the operation data processing module is used to: perform anomaly detection and risk prediction on the display equipment according to the operation data to obtain a first monitoring result; the display data processing module is used to: adopt multiple monitoring strategies that correspond one-to-one to the multimodal display data, process the multimodal display data, and obtain a second monitoring result.

[0135] In some optional implementations of this embodiment, the operation data processing module is further used to: perform anomaly detection on the display equipment according to the operation data in combination with preset detection rules and anomaly detection models to obtain anomaly detection results; perform risk prediction on the display equipment according to the operation data through multiple risk prediction models to obtain risk prediction results; and obtain a first monitoring result by combining the anomaly detection results and the risk prediction results.

[0136] In some optional implementations of the present embodiment, the multimodal display data includes sound data, the display data processing module includes a sound processing sub-module; and the sound processing sub-module is used to: perform short-time Fourier transform on the sound data to determine the energy distribution information of the sound data in the time domain and the frequency domain; determine the sound coherence of the sound data based on the energy distribution information; determine the Mel-frequency cepstral coefficients of the sound data; and determine the sound saturation of the sound data based on the Mel-frequency cepstral coefficients.

[0137] In some optional implementations of the present embodiment, the multimodal display data includes image data, the display data processing module includes an image processing sub-module; and the image processing sub-module is used to: use a perceptual hash algorithm to compare the image data with material images in a preset material library; in response to the comparison result indicating that the image data exceeds the image coverage of the preset material library, call the image sensitive detection interface to determine the compliance of the image data.

[0138] In some optional implementations of the present embodiment, the multimodal display data includes text data, the display data processing module includes a text processing submodule; and the text processing submodule is used to: identify the text represented by the sound data and the text in the image data to obtain the sound text data and the image text data; call the semantic analysis interface to determine whether the sound text data and the image text data include sensitive data respectively; and determine whether the sound text data and the image text data subtitle text data are synchronized in terms of timing.

[0139] In some optional implementations of this embodiment, the multimodal display data includes video data, the display data processing module includes a video processing submodule; and the video processing submodule is used to: based on a preset time interval, collect key frame images from the video data to obtain a key frame sequence; use a perceptual hashing algorithm to determine the similarity between adjacent key frame images in the key frame sequence; and determine the video coherence of the video data based on the similarity.

[0140] In some optional implementations of this embodiment, the above system also includes: a message queue (not shown in the figure), which is used to: transmit operation data and multimodal display data between the monitoring client and the data processing center; and transmit exception handling tasks between the exception handling center and the monitoring client.

[0141] In the present embodiment, a monitoring system for display equipment is provided, which performs monitoring in the equipment dimension and the content dimension based on the operation data and multimodal display data of the display equipment, thereby improving the comprehensiveness of the monitoring process; adopts multiple monitoring strategies that correspond one-to-one to the multimodal display data to process the multimodal display data, thereby improving the pertinence and accuracy of the content dimension monitoring process; in response to abnormal situations in the equipment dimension and / or the content dimension, determines the exception handling task corresponding to the abnormal situation, so as to handle the abnormal situation in a targeted manner, thereby reducing the intensity of manual monitoring while improving the safety and reliability of the display process.

[0142] Continue to refer Figure 6 , showing a timing diagram inside the monitoring system of the display device.

[0143] 1. Start the monitoring client and send an authentication request to the communication gateway.

[0144] 2. The communication gateway responds with the authentication result, and the monitoring client establishes an identity authentication data path with the communication gateway.

[0145] 3. The monitoring client collects the operating data and multimodal display data of the display device during the data display process, and sends the operating data and multimodal display data to the communication gateway with authentication information.

[0146] 4. After the communication gateway completes the request identity authentication, it pushes the operation data and multimodal display data to the bidirectional data message queue for cache.

[0147] 5. The processing nodes of the data processing center pull operation data and multi-modal display data based on their own load conditions.

[0148] 6. At the processing node, the operation data is processed by the operation data processing module to obtain a first monitoring result; and the multimodal display data is processed by the display data processing module to obtain a second monitoring result.

[0149] 7. Send the first monitoring result and the second monitoring result to the dispatching center.

[0150] 8. In response to the first monitoring result and / or the second monitoring result indicating that there is an abnormality in the data display process, the exception handling center arranges multiple exception handling strategies corresponding to the multiple abnormal situations based on the intelligent arrangement module to obtain an exception handling task.

[0151] 9. In response to the failure of the policy orchestration engine to orchestrate multiple exception handling policies, a manual orchestration module is used to orchestrate multiple exception handling policies to obtain an exception handling task.

[0152] 10. Cache the exception handling task in the message queue.

[0153] 11. The message queue sends the sorted exception handling tasks to the communication gateway.

[0154] 12. The communication network feeds back the sorted exception handling tasks to the monitoring client;

[0155] 13. The monitoring client controls the monitoring terminal to perform exception handling tasks.

[0156] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the monitoring method of the display device described in any of the above embodiments can be implemented when the at least one processor executes.

[0157] According to an embodiment of the present disclosure, the present disclosure further provides a readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to implement the monitoring method of a display device described in any of the above embodiments when executed.

[0158] An embodiment of the present disclosure provides a computer program product. When the computer program is executed by a processor, the monitoring method of the display device described in any of the above embodiments can be implemented.

[0159] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0160] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0161] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0162] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as the monitoring method of the display device. For example, in some embodiments, the monitoring method of the display device may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the monitoring method of the display device described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute the monitoring method of the display device in any other appropriate manner (e.g., by means of firmware).

[0163] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0164] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable display device's monitoring system, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0165] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0166] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0167] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0168] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services; it may also be a server for a distributed system, or a server combined with a blockchain.

[0169] According to the technical solution of the embodiments of the present disclosure, a monitoring method and system for display equipment are provided. Based on the operation data and multimodal display data of the display equipment, monitoring is performed in the device dimension and the content dimension, thereby improving the comprehensiveness of the monitoring process. Multiple monitoring strategies corresponding to the multimodal display data are adopted to process the multimodal display data, thereby improving the pertinence and accuracy of the content dimension monitoring process. In response to abnormal situations in the device dimension and / or the content dimension, an exception handling task corresponding to the abnormal situation is determined to handle the abnormal situation in a targeted manner, thereby reducing the intensity of manual monitoring while improving the safety and reliability of the display process.

[0170] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution provided by this disclosure can be achieved, and this document does not limit this.

[0171] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for monitoring a display device, comprising: Obtaining the operation data and multi-modal display data of the display device during the data display process; Monitor the operating status of the display device according to the operating data to obtain a first monitoring result; Adopting a plurality of monitoring strategies corresponding one to one to the multimodal display data, processing the multimodal display data, and obtaining a second monitoring result; In response to the first monitoring result and / or the second monitoring result indicating that an abnormal situation exists in the data display process, an abnormality handling task corresponding to the abnormal situation is determined and executed through the display device.

2. The method according to claim 1, wherein: In response to the first monitoring result and / or the second monitoring result indicating that an abnormality exists in the data display process, determining an abnormality handling task corresponding to the abnormality includes: In response to the first monitoring result and / or the second monitoring result indicating that the data display process has multiple abnormal situations, determining multiple abnormality handling strategies corresponding to the multiple abnormal situations; The plurality of exception handling strategies are orchestrated through a policy orchestration engine to determine the exception handling task.

3. The method according to claim 2, wherein: In response to the first monitoring result and / or the second monitoring result indicating that an abnormality exists in the data display process, determining an abnormality handling task corresponding to the abnormality further includes: In response to the policy orchestration engine failing to orchestrate the multiple exception handling policies, sending an orchestration task of the multiple exception handling policies to a target terminal; Based on the target terminal, the exception handling task is acquired.

4. The method according to claim 1, wherein: Also includes: Determining a target processing node from the data processing center based on load data of the processing nodes in the data processing center; as well as The step of monitoring the operating state of the display device according to the operating data to obtain a first monitoring result includes: By means of the operation data processing module in the target processing node, abnormality detection and risk prediction are performed on the display device according to the operation data to obtain the first monitoring result; and The adopting of the plurality of monitoring strategies corresponding to the multimodal display data one by one, processing the multimodal display data, and obtaining a second monitoring result includes: The multimodal display data is processed by the display data processing module in the target processing node using a plurality of monitoring strategies corresponding one-to-one to the multimodal display data to obtain a second monitoring result.

5. The method according to claim 4, wherein: The performing abnormality detection and risk prediction on the display device according to the operation data to obtain the first monitoring result includes: In combination with the preset detection rules and the anomaly detection model, an anomaly detection is performed on the display device according to the operation data to obtain an anomaly detection result; Perform risk prediction on the display equipment according to the operation data using multiple risk prediction models to obtain risk prediction results; The first monitoring result is obtained by combining the abnormality detection result and the risk prediction result.

6. The method according to claim 4, wherein: The multimodal presentation data includes sound data; as well as The adopting a plurality of monitoring strategies corresponding one-to-one to the multimodal display data to process the multimodal display data includes: Performing short-time Fourier transform on the sound data to determine energy distribution information of the sound data in the time domain and the frequency domain; determining the sound coherence of the sound data according to the energy distribution information; Determining Mel-frequency cepstral coefficients of the sound data; The sound saturation of the sound data is determined according to the Mel-frequency cepstral coefficients.

7. The method according to claim 6, wherein: The multimodal presentation data includes image data; and The adopting a plurality of monitoring strategies corresponding one-to-one to the multimodal display data to process the multimodal display data includes: Using a perceptual hash algorithm, the image data is compared with material images in a preset material library; In response to the comparison result indicating that the image data exceeds the image coverage of the preset material library, an image sensitive detection interface is called to determine the compliance of the image data.

8. The method according to claim 7, wherein: The multimodal presentation data includes text data; and The adopting a plurality of monitoring strategies corresponding one-to-one to the multimodal display data to process the multimodal display data includes: Identify the text represented by the sound data and the text in the image data to obtain sound text data and image text data; Calling a semantic analysis interface to determine whether the sound text data and the image text data include sensitive data; In terms of timing, it is determined whether the audio text data and the subtitle text data in the image text data are synchronized.

9. The method according to any one of claims 4 to 8, wherein: The multimodal presentation data includes video data; and The adopting a plurality of monitoring strategies corresponding one-to-one to the multimodal display data to process the multimodal display data includes: Based on a preset time interval, collecting key frame images from the video data to obtain a key frame sequence; Using a perceptual hashing algorithm, determining the similarity between adjacent key frame images in the key frame sequence; Video coherence of the video data is determined based on the similarity.

10. The method according to any one of claims 1 to 9, wherein: The obtaining of the operation data and multimodal display data of the display device during the data display process includes: Obtaining the operation data and the multimodal display data from a message queue; and The performing of the exception handling task corresponding to the abnormal situation by the display device includes: Transmitting the exception handling task to the message queue; The exception handling task in the message queue is executed through the display device.

11. A monitoring system for display equipment, comprising: Monitoring client, used to collect the operation data and multi-modal display data of the display device during the data display process; A data processing center is used to monitor the operating status of the display device according to the operating data to obtain a first monitoring result; Adopting a plurality of monitoring strategies corresponding one to one to the multimodal display data, processing the multimodal display data, and obtaining a second monitoring result; an exception handling center, configured to determine an exception handling task corresponding to an abnormal situation in response to the first monitoring result and / or the second monitoring result indicating that an abnormal situation exists in the data display process; The monitoring client is also used to control the display device to execute the exception handling task.

12. The system according to claim 11, wherein: The exception handling center includes an intelligent orchestration module, and The intelligent arrangement module is used to: In response to the first monitoring result and / or the second monitoring result indicating that the data display process has multiple abnormal situations, determining multiple abnormality handling strategies corresponding to the multiple abnormal situations; The plurality of exception handling strategies are orchestrated through a policy orchestration engine to determine the exception handling task.

13. The method according to claim 2, wherein: The exception handling center includes a manual arrangement module, and The manual arrangement module is used to: In response to the policy orchestration engine failing to orchestrate the multiple exception handling policies, sending an orchestration task of the multiple exception handling policies to a target terminal; Based on the target terminal, the exception handling task is acquired.

14. The method according to claim 11, wherein: The data processing center is also used for: According to the load data of the processing nodes in the data processing center, a target processing node is determined from the data processing center, wherein the target processing node includes a running data processing module and a display data processing module, and The operation data processing module is used to: Performing abnormality detection and risk prediction on the display device according to the operation data to obtain the first monitoring result; The display data processing module is used to: A plurality of monitoring strategies corresponding one to one to the multimodal display data are adopted to process the multimodal display data to obtain a second monitoring result.

15. The system of claim 14, wherein: The operation data processing module is further used for: In combination with the preset detection rules and the anomaly detection model, the display device is subjected to an anomaly detection according to the operation data to obtain an anomaly detection result; and the risk prediction of the display device is performed according to the operation data by using multiple risk prediction models to obtain a risk prediction result; The first monitoring result is obtained by combining the abnormality detection result and the risk prediction result.

16. The method according to claim 14, wherein: The multimodal presentation data includes sound data, and the presentation data processing module includes a sound processing submodule; and The sound processing submodule is used for: Performing a short-time Fourier transform on the sound data to determine energy distribution information of the sound data in the time domain and the frequency domain; determining the sound coherence of the sound data based on the energy distribution information; determining the Mel-frequency cepstral coefficients of the sound data; and determining the sound saturation of the sound data based on the Mel-frequency cepstral coefficients.

17. The system of claim 16, wherein: The multimodal presentation data includes image data, and the presentation data processing module includes an image processing submodule; and The image processing submodule is used for: A perceptual hash algorithm is used to compare the image data with material images in a preset material library; in response to the comparison result indicating that the image data exceeds the image coverage of the preset material library, an image sensitive detection interface is called to determine the compliance of the image data.

18. The system of claim 17, wherein: The multimodal presentation data includes text data, and the presentation data processing module includes a text processing submodule; and The text processing submodule is used to: Identify the text represented by the sound data and the text in the image data to obtain sound text data and image text data; call a semantic analysis interface to determine whether the sound text data and the image text data include sensitive data; and determine whether the sound text data and the subtitle text data in the image text data are synchronized in terms of timing.

19. The system according to any one of claims 14 to 18, wherein: The multimodal presentation data includes video data, and the presentation data processing module includes a video processing submodule; and The video processing submodule is used for: Based on a preset time interval, key frame images are collected from the video data to obtain a key frame sequence; a perceptual hash algorithm is used to determine the similarity between adjacent key frame images in the key frame sequence; and video coherence of the video data is determined based on the similarity.

20. The system according to any one of claims 11 to 19, wherein: Also includes: Message queues, used for: Transmitting the operation data and the multimodal display data between the monitoring client and the data processing center; The exception handling task is transmitted between the exception handling center and the monitoring client.

21. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 10.

23. A computer program product comprising: A computer program which, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images