Media resource data detection method and device, electronic equipment and storage medium

Abstract

Embodiments of the present disclosure provide a media resource data detection method and device, electronic equipment and a storage medium. The method comprises determining target real-time media resource data associated with a target data full link, wherein the target data full link is a data link used when a target media resource demand event is triggered; detecting task delay time when the target real-time media resource data sequentially passes through each layer of task nodes in the target data full link to perform a data processing task; and determining a data quality detection result of the target real-time media resource data according to the task delay time in the target data full link, wherein the data quality detection result is used to describe the performance and availability of the target real-time media resource data. The present scheme solves the problems of ambiguous definition and insufficient quantification of massive real-time media resource data quality, and quantifies the real-time media resource data quality in a timely manner to ensure the timeliness and accuracy of the real-time media resource data as much as possible.

Description

Technical Field

[0001] This disclosure relates to the field of data technology, and in particular to a method, apparatus, electronic device, and storage medium for detecting media resource data. Background Technology

[0002] Real-time processing of massive amounts of real-time media resource data is a common scenario in big data applications. This data is characterized by its large volume and high requirements for timeliness and accuracy, and is widely used in various real-time data reports. However, when providing reliable real-time media resource data to users, quality issues may exist. If these issues are not detected in time, the real-time media resource data may fail to meet timeliness and accuracy requirements, thus severely impacting its usability. Summary of the Invention

[0003] This disclosure provides a method, apparatus, electronic device, and storage medium for detecting media resource data, in order to solve the problem of insufficient data quality quantification of real-time media resource data and to achieve effective detection of real-time media resource data.

[0004] In a first aspect, embodiments of this disclosure provide a method for detecting media resource data, the method comprising:

[0005] Determine the target real-time media resource data associated with the entire target data link, wherein the entire target data link is the data link used when the target media resource demand event is triggered;

[0006] The task delay time is detected when the target real-time media resource data sequentially passes through each layer of task nodes in the target data full link to perform data processing tasks.

[0007] The data quality detection result of the target real-time media resource data is determined based on the task latency time of the entire target data link. The data quality detection result is used to describe the performance and availability of the target real-time media resource data.

[0008] Secondly, this disclosure also provides a media resource data detection device, the device comprising:

[0009] The determination module is used to determine the target real-time media resource data associated with the entire target data link, wherein the entire target data link is the data link used when the target media resource demand event is triggered.

[0010] The first detection module is used to detect the task delay time when the target real-time media resource data sequentially passes through each layer of task nodes in the target data full link to perform data processing tasks.

[0011] The second detection module is used to determine the data quality detection result of the target real-time media resource data based on the task delay time in the entire target data link. The data quality detection result is used to describe the performance and availability of the target real-time media resource data.

[0012] Thirdly, embodiments of this disclosure also provide an electronic device, the electronic device comprising:

[0013] One or more processors;

[0014] Storage device for storing one or more programs.

[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement the media resource data detection method according to any embodiment of this disclosure.

[0016] Fourthly, embodiments of this disclosure also provide a storage medium containing computer-executable instructions, wherein the computer-readable storage medium stores computer instructions, which, when executed by a computer processor, are used to perform the media resource data detection method described in any embodiment of this disclosure.

[0017] This embodiment of the disclosure acquires target real-time media resource data associated with the entire target data chain. By detecting the task delay time when the target real-time media resource data sequentially passes through each task node in the target data chain to perform data processing tasks, and quantifying the data quality of the target real-time media resource data based on the task delay time in the target data chain, the performance and availability of the target real-time media resource data are described. This solves the problems of fuzzy definition and insufficient quantification of the quality of massive real-time media resource data. By quantifying the quality of real-time media resource data in a timely manner, the timeliness and accuracy of real-time media resource data are guaranteed as much as possible, providing a guarantee for providing reliable real-time media resource data to users. Attached Figure Description

[0018] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0019] Figure 1 This is a flowchart illustrating a media resource data detection method provided in an embodiment of this disclosure;

[0020] Figure 2 This is a general schematic flowchart illustrating a real-time media resource data quality quantification method provided in this embodiment of the disclosure;

[0021] Figure 3 This is a detailed schematic flowchart illustrating a real-time media resource data quality quantification method provided in this embodiment of the disclosure;

[0022] Figure 4 This is a flowchart illustrating another media resource data detection method provided in this embodiment of the disclosure;

[0023] Figure 5 This is a schematic diagram illustrating the statistical duration of data unavailability across the entire data link, provided in an embodiment of this disclosure.

[0024] Figure 6 This is a different data end-to-end task latency trend provided by the embodiments of this disclosure;

[0025] Figure 7 This is a schematic diagram of the structure of a media resource data detection device provided in an embodiment of this disclosure;

[0026] Figure 8 This is a schematic diagram of the structure of an electronic device that implements the media resource data detection method according to an embodiment of this disclosure. Detailed Implementation

[0027] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0028] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0029] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0030] It should be noted that the concepts of "first," "second," "objective," and "reference" mentioned in this disclosure are used only to distinguish different devices, modules, or units, and are not used to define the order of functions performed by these devices, modules, or units or their interdependencies.

[0031] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0032] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0033] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0034] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0035] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0036] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0037] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0038] Figure 1 This is a flowchart of a media resource data detection method provided in an embodiment of the present disclosure. This embodiment of the present disclosure is applicable to the quantitative detection of real-time media resource data in a big data system. The method can be executed by a media resource data detection device, which can be implemented in the form of software and / or hardware, and can be implemented by configuring it on any electronic device with network communication function, such as a mobile terminal, a PC, or a server.

[0039] like Figure 1As shown, the media resource data detection in this embodiment may include the following process:

[0040] S110. Determine the target real-time media resource data associated with the target data throughout the entire link. The target data throughout the entire link is the data link used when the target media resource demand event is triggered.

[0041] Real-time processing of massive amounts of real-time media resource data is a common scenario in big data applications. Real-time media resource data is characterized by its large volume, high timeliness, and accuracy requirements, and is widely used in various real-time data reports. When there is a demand for a target media resource, a target media resource demand event can be triggered. In this case, the data link required to fulfill the target media resource demand event can be considered as the target data end-to-end. For example, when there is a demand for advertising information resources, an advertising information resource demand event can be triggered by clicking the advertising resource icon displayed on the page. In this case, the data link corresponding to the advertising information resource demand event is needed to provide the advertising resource data.

[0042] Target real-time media resource data can be reliable real-time media resource data provided through triggering target media resource demand events and utilizing the entire target data chain. Real-time media resource data can include, but is not limited to, the following forms of media resource data: news resource data, novel resource data, social information resource data, video resource data, and advertising information resource data.

[0043] S120. Determine the task delay time when the target real-time media resource data sequentially passes through each layer of task nodes in the target data full link to perform data processing tasks.

[0044] After a target media resource demand event is triggered, reliable real-time target media resource data corresponding to the target media resource demand event can be provided through the target data end-to-end. The target data end-to-end includes multiple layers of task nodes, each associated with a preset data processing task. In the process of providing target real-time media resource data through the target data end-to-end, the target data end-to-end will sequentially reach each layer of task nodes in the target data end-to-end to execute the corresponding data processing task logic.

[0045] Due to the high timeliness of real-time media resource data, the latency of the target real-time media resource data throughout the entire data chain can be used as a quantification indicator of its quality. In the real-time media resource data of the big data system, the time of triggering the target media resource demand event corresponding to the target real-time media resource data is added, utilizing the timing of the media resource demand event to connect the task nodes at each layer of the entire data chain.

[0046] Furthermore, the time difference between the execution time of the corresponding data processing task at each task node and the time when the target media resource demand event is triggered can be determined, thus obtaining the task delay time of the target real-time media resource data up to the current execution of the task in the entire target data chain. The data processing tasks executed by each task node in the entire target data chain can include one or more of the following: data cleaning, data transformation, data extraction, and data computation.

[0047] As an optional but not limited implementation, determining the task latency time when the target real-time media resource data sequentially passes through each task node in the target data chain to perform data processing tasks may include the following steps A1-A2:

[0048] Step A1: Determine the target delay time data of the target real-time media resource data in the target data end link from the preset storage database. The target delay time data is collected, reported and stored in the preset storage database by task nodes at each layer in the target data end link using a unified preset interface. The preset interface is configured using a software development kit.

[0049] Step A2: From the target delay time data, determine the task delay time of each task node based on the target real-time media resource data. The task delay time is the time difference between the actual time and the reference time when the data processing task is executed at the task node. The reference time is the time when the corresponding event of the target real-time media resource data is triggered.

[0050] See Figure 2 This paper presents a schematic flowchart for quantifying the quality of real-time media resource data. This application proposes to use the task latency time of the entire data link as an indicator to measure the quality of real-time media resource data. In order to achieve quantitative analysis of data quality, the application adopts the following steps: target real-time media resource data calculation, target real-time media resource data reporting, target real-time media resource data collection, and target real-time media resource data statistics, so as to quantify the task latency time of each task node in the entire data link.

[0051] See Figure 2 and Figure 3 At each node of the target data chain, the target latency of the real-time media resource data can be obtained by using the actual time of data processing tasks executed at each task node and the time when the corresponding event triggers the target real-time media resource data. To obtain the target latency data obtained from data processing tasks at each task node for subsequent unified processing, the target latency data calculated at each task node can be reported to the database. The data processing tasks executed at each task node can be real-time Flink tasks.

[0052] Optionally, see Figure 2 and Figure 3 To achieve unified reporting of target latency data obtained from data processing tasks executed at each task node, a unified software development kit (SDK) is used to configure the data processing tasks at each task node. Then, each task node can use the preset interface configured by the SDK to send the target latency data calculated at each task node to a remote storage database for storage. At the same time, the data processing task identifier (such as task name) corresponding to the target latency data and the full data link to which the target latency data belongs are attached when storing the data, so as to make targeted selections when used later.

[0053] Optionally, see Figure 3 After the target latency data is reported to a unified database, the reported target latency data can be collected periodically at a preset frequency through a scheduled service. The target latency data is then preprocessed and stored in the offline data warehouse tool Hive database as a data source for subsequent complex data statistics.

[0054] As an optional but not limited implementation method, the target real-time media resource data is selected from the real-time media resource data processed throughout the target data chain based on a reference sampling percentage, which is determined based on the different orders of magnitude of the real-time media resource data processed throughout the target data chain.

[0055] See Figure 3 In big data systems, the volume of real-time media resource data is enormous, resulting in a vast amount of real-time media resource data undergoing processing throughout the entire target data chain. Calculating the latency for each piece of real-time media resource data throughout the entire target data chain could lead to performance degradation during data processing tasks. Therefore, this application employs a random sampling method, selecting a certain reference sampling percentage based on the different orders of magnitude of the real-time media resource data processed throughout the target data chain, and then selecting target real-time media resource data from this data according to the sampling percentage.

[0056] As an optional but not limited implementation method, each task node in the target data end-to-end encapsulates the target structured query language logic. The target structured query language logic is used to calculate and report the task delay time when the task node performs data processing tasks based on the target real-time media resource data.

[0057] See Figure 3At each task node in the entire target data chain, the task latency calculation logic based on the target real-time media resource data can be encapsulated into a general target structured query language (SQL) logic. This SQL logic can then be embedded into the data processing tasks of each task node, allowing the general target structured query language logic to calculate the task latency of each task node.

[0058] S130. Determine the data quality detection result of the target real-time media resource data based on the task delay time of the target data in the entire link. The data quality detection result is used to describe the performance and availability of the target real-time media resource data.

[0059] The task delay time of the target real-time media resource data in the target data full link includes the first task delay time or the second task delay time. The first task delay time can be the task delay time at each task node when the target real-time media resource data sequentially passes through each task node in the target data full link to execute data tasks. The second task delay time can be the sum of the task delay times when the target real-time media resource data executes data tasks at each task node.

[0060] After determining the task delay time of the target real-time media resource data throughout the entire target data chain, the task delay time can be used to quantitatively analyze the data delay of the real-time media resource data throughout the entire target data chain. Quantitative analysis of the data delay throughout the target data chain reflects the quality of real-time media resource data, which can solve the problem of the fuzzy definition and inability to quantify the quality of real-time media resource data.

[0061] The availability metrics for real-time media resource data are defined based on the task latency time across the entire target data chain, serving as the SLA (Service-Level Agreement) metric. An SLA refers to reaching an agreement on data quality at the system metric level. Compared to offline SLA definitions, real-time media resource data SLAs are more complex and difficult to define, but they are indispensable as a standard for evaluating real-time data quality and a key indicator for externally demonstrating data quality. SLAs also need to be quantifiable. This case calculates relevant SLA metrics based on the task latency time across the entire target data chain, and then uses the calculated data quality detection results to determine whether the preset SLA requirements are met.

[0062] This embodiment of the disclosure can acquire target real-time media resource data associated with the entire data chain. By detecting the task delay time when the target real-time media resource data sequentially passes through each task node in the target data chain to perform data processing tasks, and quantifying the data quality of the target real-time media resource data based on the task delay time in the target data chain, the performance and availability of the target real-time media resource data are described. Based on the core field of the task delay time in the entire data chain of real-time media resource data, the task nodes in each layer of the chain are connected, and a unified process of task delay time calculation, reporting, collection, statistics, and visualization is completed. This solves the problem of fuzzy definition and insufficient quantification of the quality of massive real-time media resource data. By quantifying the quality of real-time media resource data in a timely manner, the timeliness and accuracy of real-time media resource data are guaranteed as much as possible, providing a guarantee for providing reliable real-time media resource data to users.

[0063] Figure 4 This disclosure provides a flowchart of another media resource data detection method. The technical solution of this embodiment further optimizes the process of determining the data quality detection result of the target real-time media resource data based on the task delay time of the target data in the above embodiment, based on the above embodiment. This embodiment can be combined with various optional solutions in one or more of the above embodiments.

[0064] like Figure 4 As shown, the media resource data detection method in this embodiment may include the following process:

[0065] S410. Determine the target real-time media resource data associated with the target data throughout the entire data link. The target data throughout the entire data link is the data link used when the target media resource demand event is triggered.

[0066] S420, Determine the task delay time when the target real-time media resource data sequentially passes through each layer of task nodes in the target data full link to perform data processing tasks.

[0067] S430. Based on the task delay time in the target data end link, determine the target duration of the target data end link. The target duration is the duration during which the target data end link is in an unavailable state when the task nodes in the target data end link are performing data processing tasks.

[0068] See Figure 5When a failure occurs in the entire target data link, delays will occur in data processing tasks at each task node in the target data link, resulting in task latency. When the task latency at each task node is long, the availability of that task node in the target data link can be considered poor, or even directly considered to be in an unusable state. Therefore, the duration of the target data link being unavailable while task nodes in the target data link are performing data processing tasks can be calculated based on the task latency at each task node in the target data link.

[0069] The reason for statistically analyzing the duration of the target data being unavailable across the entire data processing chain at each task node is that real-time data processing tasks for real-time media resource data at each task node often experience momentary stability fluctuations due to transient failures in the network, dependent components, etc. Some of these fluctuations may cause brief delays in the processing of real-time media resource data, but these recover quickly on their own and are essentially imperceptible. Simply using the delay time at each task node to measure data quality would introduce a lot of irrelevant noise, leading to inconsistencies between data quality detection results and actual user experience. Therefore, statistically analyzing the duration of the target data being unavailable across the entire chain is a better way to measure the data quality of real-time media resource data.

[0070] As an optional but not limited implementation, determining the target duration of the entire target data link based on the task latency time across the entire target data link can include steps B1-B3:

[0071] Step B1: Based on the task delay time of the entire target data link, determine the start time point when the entire target data link is in an unavailable state. The start time point is the time point when the task delay time of the entire target data link begins to rise and exceeds the preset delay time threshold.

[0072] Step B2: Based on the task delay time of the entire target data link, determine the end time point when the entire target data link is in an unavailable state. The end time point is the time point when the task delay time of the entire target data link begins to decrease and is less than the preset delay time threshold.

[0073] Step B3: Determine the target duration of the entire target data link based on the start and end times.

[0074] See Figure 5Taking the target data link as data link A as an example, for the target data link, it can be defined that if the task delay time of the target data link exceeds the preset delay time threshold, the target data link is considered to become unavailable; and if the task delay time of the target data link is lower than the preset delay time threshold, the target data link is considered to become available. In this way, the duration of the target data link being in an unavailable state can be counted.

[0075] See Figure 5 When a failure occurs in the target data end link, the task delay time of the target data end link begins to increase. When the task delay time of the target data end link exceeds the preset delay time threshold, the target data end link becomes unavailable. The time point when the target data end link just becomes unavailable can be recorded as the start time point.

[0076] See Figure 5 When a failure occurs in the target data link and fault recovery begins, the target data link remains unavailable as long as the task latency begins to decrease and does not fall below a preset latency threshold. Only when the task latency continuously decreases below the preset latency threshold will the target data link become available. The time point at which the target data link changes from unavailable to available can be recorded as the end time point. Furthermore, the duration from the start time point to the end time point can be calculated as the target duration of the target data link.

[0077] For example, see Figure 5 Taking data link A as an example, the threshold for the task latency of data link A is 30 minutes. When link A fails, the task latency of the entire link will increase due to the impaired processing performance. When it exceeds 30 minutes, link A is considered to be unavailable. The time when link A becomes unavailable is set as the start time. After the failure of link A is repaired, its processing performance recovers, and the task latency of the entire link will decrease. When it is less than 30 minutes, link A is considered to be available. The time when link A becomes available is set as the end time. The duration between the start time and the end time is calculated as the unavailable duration of the entire target data link.

[0078] S440. Determine the data quality detection result of the target real-time media resource data based on the target duration of the entire target data link. The data quality detection result is used to describe the performance and availability of the target real-time media resource data.

[0079] As an optional but not limited implementation method, determining the data quality detection result of the target real-time media resource data based on the target duration of the entire target data link may include steps C1-C3:

[0080] Step C1: Determine the target duration of the entire target data link within a single statistical period from the target duration of the entire target data link.

[0081] Step C2: Determine the target percentage based on the total duration of a single statistical period and the target duration within the single statistical period. The target percentage is the percentage of time during which the target data is unavailable across the entire link within a single statistical period.

[0082] Step C3: Determine the data quality detection results of the target real-time media resource data within a single statistical period based on the target proportion.

[0083] See Figure 5 If we simply use the latency of each task node to measure data quality, it will introduce a lot of unnecessary noise, resulting in data quality detection results that are inconsistent with the actual user experience. Therefore, based on the duration of the target data being unavailable throughout the entire link, we divide the statistical period according to different statistical times. Then, we can calculate the duration of the target data being unavailable throughout the entire link in each statistical period, and then determine the SLA value by statistically analyzing the proportion of time the target data is unavailable throughout the entire link in a single statistical period, so as to obtain the data quality detection results of the target real-time media resource data. See Table 1 for the SLA statistics of different data links.

[0084] Table 1. SLA statistics for different data chains

[0085] SLA benchmark SLA percentage (%) Duration (minutes) unavailable Maximum link latency (minutes) 98 98.16 25 70.1000 98 98.89 15 86.05000 98 98.89 15 47.6167 98 99.26 10 40.6833

[0086] Optionally, the SLA calculation formula for a single statistical period is as follows:

[0087]

[0088] Wherein, SLA represents the performance and availability of the target real-time media resource data, A1 represents the duration during which the target data is unavailable in a single statistical period, and A2 represents the total duration of a single statistical period.

[0089] As an optional but not limited implementation, after determining the data quality inspection results of the target real-time media resource data, it may also include:

[0090] Based on the data quality detection results of the target real-time media resource data corresponding to each statistical period, the trend information of the change in the real-time media resource data quality under the entire target data link is determined.

[0091] The information on the changing trend of real-time media resource data quality is used to describe the changes in the performance and availability of the target real-time media resource data in the entire target data processing chain over time. If the performance and availability of the target real-time media resource data gradually become satisfactory, it indicates that the faults of each task node in the entire target data processing chain are slowly recovering, and we only need to wait for recovery. Based on the performance and availability of the target real-time media resource data, we can ensure the stability of the task and prevent the fault duration from increasing. If the performance and availability of the target real-time media resource data do not change in the direction of gradually becoming satisfactory, it indicates that the faults of each task node in the entire target data processing chain have not been alleviated. At this time, it is necessary to optimize each task node to optimize the MTTR (Mean Time To Restoration) as much as possible to ensure that the faults in the entire target data processing chain can be recovered quickly.

[0092] As an optional but not limited implementation, after determining the data quality inspection results of the target real-time media resource data, it may also include:

[0093] Based on the data quality inspection results of the target real-time media resource data, a visual report on the real-time media resource data processing of the entire target data chain is generated and pushed out for display.

[0094] Based on the performance and availability of the target real-time media resource data, the real-time media resource data is processed in depth to obtain a data structure that can be visualized and displayed, and finally put into production use to serve the visualization report for media resource data quality monitoring. In addition, the real-time media resource data can be embedded for multi-angle monitoring based on the performance and availability of the target real-time media resource data.

[0095] As an optional but not limited implementation, after determining the data quality inspection results of the target real-time media resource data, it may also include:

[0096] Based on the data quality detection results of the target real-time media resource data, the performance of each task node in the target data end link is optimized to enhance the recovery speed after data delay occurs in the target end link.

[0097] Based on the performance and availability of the target real-time media resource data, the stability of the task is guaranteed and the downtime is reduced. At the same time, the task nodes at each layer are optimized based on the performance and availability of the target real-time media resource data to optimize MTTR as much as possible, so as to ensure that the target data can be recovered quickly in the event of a failure across the entire link.

[0098] As an optional but not limited implementation, the media resource data detection in this embodiment may further include the following steps, D1-D2.

[0099] Step D1: Based on the task delay time in the entire target data link, determine the task delay time of different target real-time media resource data associated with each task node in the entire target data link.

[0100] Step D2: Based on the task latency time of the different target real-time media resource data associated with each task node, generate task latency time change trend information for each task node, in order to optimize the performance of each task node.

[0101] For target real-time media resource data processed at different times throughout the entire target data chain, the task latency time at each task node can be obtained as a function of time. Using the task latency times of different target real-time media resource data associated with each task node, a trend of task latency time variation over time for each task node can be generated. Furthermore, by combining the trend of task latency time variation over time for each layer of task nodes, the trend of task latency time variation over time for the target data chain to which each layer of task nodes belongs can be obtained. (See [reference needed]). Figure 6 The diagram shows how the task latency of different data links changes over time.

[0102] This embodiment of the disclosure, based on the core field of task latency time in the entire data link of real-time media resource data, connects the task nodes at each layer of the link, completing a unified process of task latency time calculation, reporting, collection, statistics, and visualization. This solves the problems of ambiguous definition and insufficient quantification of the quality of massive real-time media resource data. By quantifying the quality of real-time media resource data in a timely manner, it ensures the timeliness and accuracy of real-time media resource data as much as possible, providing a guarantee for providing reliable real-time media resource data to users. Furthermore, unlike statistical methods based on fixed time points, it innovatively defines a real-time SLA quantification calculation based on data availability and provides detailed statistical rules for real-time SLA, incorporating MTTR into the real-time data quality model, while eliminating data noise caused by occasional negligible latency due to network jitter.

[0103] Figure 7 This is a structural diagram of a media resource data detection device provided in an embodiment of the present disclosure. This embodiment of the present disclosure is applicable to the quantitative detection of real-time media resource data in a big data system. The device can be implemented in the form of software and / or hardware, and can be implemented by configuring it on any electronic device with network communication function, such as a mobile terminal, PC, or server.

[0104] like Figure 7As shown, the media resource data detection device of this embodiment may include: a determination module 710, a first detection module 720, and a second detection module 730. Wherein:

[0105] The determination module 710 is used to determine the target real-time media resource data associated with the entire target data link, wherein the entire target data link is the data link used when the target media resource demand event is triggered.

[0106] The first detection module 720 is used to detect the task delay time when the target real-time media resource data sequentially passes through each layer of task nodes in the target data full link to perform data processing tasks.

[0107] The second detection module 730 is used to determine the data quality detection result of the target real-time media resource data based on the task delay time in the target data end link. The data quality detection result is used to describe the performance and availability of the target real-time media resource data.

[0108] Optionally, based on the above embodiments, the target real-time media resource data is selected from the real-time media resource data processed throughout the target data chain according to a reference sampling percentage, wherein the reference sampling percentage is determined according to different orders of magnitude of the real-time media resource data processed throughout the target data chain.

[0109] Based on the above embodiments, optionally, the task delay time when the target real-time media resource data sequentially passes through each layer of task nodes in the target data full link to perform data processing tasks includes:

[0110] The target latency time data of the target real-time media resource data in the target data full link is determined from the preset storage database. The target latency time data is collected, reported and stored in the preset storage database by task nodes at each layer in the target data full link using a unified preset interface. The preset interface is configured using a software development kit.

[0111] From the target delay time data, the task delay time of the target real-time media resource data at each task node is determined. The task delay time is the time difference between the actual time and the reference time when the data processing task is executed at the task node. The reference time is the time when the event corresponding to the target real-time media resource data is triggered.

[0112] Based on the above embodiments, optionally, each task node in the target data end-to-end chain encapsulates target structured query language logic, which is used to calculate and report the task delay time when the task node performs data processing tasks based on target real-time media resource data.

[0113] Based on the above embodiments, optionally, the data quality detection result of the target real-time media resource data is determined according to the task delay time in the entire target data link, including:

[0114] Based on the task delay time in the entire target data link, the target duration of the entire target data link is determined. The target duration is the duration during which the entire target data link is unavailable when the task nodes in the target data link are performing data processing tasks.

[0115] The data quality detection result of the target real-time media resource data is determined based on the target duration of the entire target data link.

[0116] Based on the above embodiments, optionally, the target duration of the entire target data link is determined according to the task latency time in the entire target data link, including:

[0117] Based on the task delay time of the entire target data link, the start time point when the entire target data link is in an unavailable state is determined. The start time point is the time point when the task delay time of the entire target data link begins to rise and exceeds a preset delay time threshold.

[0118] Based on the task delay time of the entire target data link, the end time point when the entire target data link is in an unavailable state is determined. The end time point is the time point when the task delay time of the entire target data link begins to decrease and is less than a preset delay time threshold.

[0119] Based on the start time and the end time, the target duration of the entire target data link is determined.

[0120] Based on the above embodiments, optionally, the data quality detection result of the target real-time media resource data is determined according to the target duration of the entire target data link, including:

[0121] From the target duration of the entire target data link, determine the target duration of the entire target data link within a single statistical period;

[0122] The target percentage is determined based on the total duration of the single statistical period and the target duration within the single statistical period. The target percentage is the percentage of time during which the target data is unavailable across the entire link within the single statistical period.

[0123] The data quality detection results of the target real-time media resource data within a single statistical period are determined based on the target proportion.

[0124] Optionally, based on the above embodiments, after determining the data quality detection result of the target real-time media resource data, the method further includes:

[0125] Based on the data quality detection results of the target real-time media resource data corresponding to each statistical period, the trend information of the change in the real-time media resource data quality under the entire target data link is determined.

[0126] Optionally, based on the above embodiments, after determining the data quality detection result of the target real-time media resource data, the method further includes:

[0127] Based on the data quality detection results of the target real-time media resource data, a visual report on the real-time media resource data processing of the entire target data chain is generated and pushed out for display.

[0128] Optionally, based on the above embodiments, after determining the data quality detection result of the target real-time media resource data, the method further includes:

[0129] Based on the data quality detection results of the target real-time media resource data, the performance of each task node in the target data end link is optimized to enhance the recovery speed after data delay occurs in the target end link.

[0130] Optionally, based on the above embodiments, the media resource data detection device of this embodiment further includes:

[0131] Based on the task latency time across the entire target data chain, determine the task latency time of different target real-time media resource data associated with each task node in the target data chain.

[0132] Based on the task latency of different target real-time media resource data associated with each task node, the task latency change trend information of each task node is generated to optimize the performance of each task node.

[0133] The media resource data detection device provided in this embodiment can execute the media resource data detection method provided in any of the above embodiments of this disclosure, and has the corresponding functions and beneficial effects of executing the media resource data detection method. For details, please refer to the relevant operations of the media resource data detection method in the foregoing embodiments.

[0134] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this disclosure.

[0135] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Reference is made below. Figure 8 It illustrates an electronic device suitable for implementing embodiments of the present disclosure (e.g., Figure 8 The diagram below shows the structure of the terminal device or server 800. The terminal device in this embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and vehicle terminals (e.g., vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0136] like Figure 8 As shown, the electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the electronic device 800. The processing device 801, ROM 802, and RAM 803 are interconnected via a bus 804. An edit / output (I / O) interface 805 is also connected to the bus 804.

[0137] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 808 including, for example, magnetic tapes, hard disks, etc.; and communication devices 809. Communication device 809 allows electronic device 800 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 8 An electronic device 800 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0138] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by a processing device 801, it performs the functions defined in the methods of embodiments of this disclosure.

[0139] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0140] The electronic device provided in this embodiment and the media resource data detection method provided in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0141] This disclosure provides a computer storage medium storing a computer program that, when executed by a processor, implements the media resource data detection method provided in the above embodiments.

[0142] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0143] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0144] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0145] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to:

[0146] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: determine target real-time media resource data associated with the target data end-to-end link, wherein the target data end-to-end link is the data link used when triggering a target media resource demand event; determine the task delay time when the target real-time media resource data sequentially passes through each layer of task nodes in the target data end-to-end link to perform data processing tasks; and determine the data quality detection result of the target real-time media resource data based on the task delay time in the target data end-to-end link, wherein the data quality detection result is used to describe the performance and availability of the target real-time media resource data.

[0147] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0148] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0149] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0150] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0151] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0152] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0153] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0154] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for detecting media resource data, characterized in that, The method comprises: determining target real-time media resource data of target data full link correlation, the target data full link being a data link adopted when a target media resource demand event is triggered; the target real-time media resource data being selected from real-time media resource data processed by the target data full link according to a reference sampling percentage, the reference sampling percentage being determined according to different orders of magnitude of the real-time media resource data processed by the target data full link; determining task delay time of the target real-time media resource data when sequentially passing through each layer task node of the target data full link to execute a data processing task; the task delay time of the target real-time media resource data when sequentially passing through each layer task node of the target data full link to execute a data processing task comprising: determining target delay time data of the target real-time media resource data in the target data full link from a preset storage database, the target delay time data being collected and reported by each layer task node of the target data full link using a unified preset interface and stored in the preset storage database, the preset interface being configured using a software development kit; determining task delay time of the target real-time media resource data based on each layer task node from the target delay time data, the task delay time being a time difference value between actual time and reference time when the task node executes a data processing task, the reference time being a time when a corresponding event of the target real-time media resource data is triggered; determining a data quality detection result of the target real-time media resource data according to the task delay time in the target data full link, the data quality detection result being used to describe performance and availability of the target real-time media resource data.

2. The method of claim 1, wherein, Each layer task node in the target data full link is encapsulated with target structured query language logic, the target structured query language logic being used to calculate and report storage of the task delay time of the target real-time media resource data when executing a data processing task based on the target real-time media resource data.

3. The method of claim 1, wherein, Determining a data quality detection result of the target real-time media resource data according to the task delay time in the target data full link comprises: determining a target duration of the target data full link according to the task delay time in the target data full link, the target duration being a duration of the target data full link being in an unavailable state when the task node executes a data processing task in the target data full link; determining a data quality detection result of the target real-time media resource data according to the target duration of the target data full link.

4. The method of claim 3, wherein, Determining a target duration of the target data full link according to the task delay time in the target data full link comprises: determining a starting time point of the target data full link being in an unavailable state according to the task delay time in the target data full link, the starting time point being a time point when the task delay time in the target data full link starts to rise and is greater than a preset delay time threshold; determining an end time point at which the target data full link is in the unavailable state according to a task delay time of the target data full link, the end time point being a time point at which the task delay time of the target data full link starts to decrease and is less than a preset delay time threshold; determining a target duration of the target data full link based on the start time point and the end time point.

5. The method of claim 3, wherein, determining a data quality detection result of the target real-time media resource data according to the target duration of the target data full link, including: determining a target duration of the target data full link in a single statistical period from the target duration of the target data full link; determining a target proportion based on a cycle total duration of the single statistical period and the target duration in the single statistical period, the target proportion being a time proportion of the target data full link in the single statistical period in the unavailable state; determining a data quality detection result of the target real-time media resource data in the single statistical period according to the target proportion.

6. The method of claim 5, wherein, After determining the data quality detection result of the target real-time media resource data, the method further includes: determining change trend information of real-time media resource data quality under the target data full link according to the data quality detection result of the target real-time media resource data corresponding to each statistical period.

7. The method according to any one of claims 1 to 6, characterized in that, After determining the data quality detection result of the target real-time media resource data, the method further includes: generating a visual report of the target data full link for processing real-time media resource data based on the data quality detection result of the target real-time media resource data and pushing and displaying the visual report.

8. The method according to any one of claims 1 to 6, characterized in that, After determining the data quality detection result of the target real-time media resource data, the method further includes: optimizing performance of each layer task node in the target data full link based on the data quality detection result of the target real-time media resource data, so as to enhance recovery speed of the target full link after data delay.

9. The method of claim 1, wherein, The method further includes: determining a task delay time of different target real-time media resource data associated with each task node in the target data full link according to a task delay time of the target data full link; generating task delay time change trend information of each task node according to the task delay time of different target real-time media resource data associated with each task node, so as to optimize performance of each task node.

10. A media resource data detection apparatus, characterized in that, The device includes: a determination module configured to determine target real-time media resource data associated with a target data full link, the target data full link being a data link used when a target media resource demand event is triggered; the target real-time media resource data being selected from real-time media resource data processed by the target data full link according to a reference sampling percentage, the reference sampling percentage being determined according to different orders of magnitude of the real-time media resource data processed by the target data full link. The first detection module is configured to detect a task delay time of the target real-time media resource data when the target real-time media resource data sequentially passes through each layer task node of the target data full link to perform a data processing task; the task delay time of the target real-time media resource data when the target real-time media resource data sequentially passes through each layer task node of the target data full link to perform a data processing task is determined by: determining target delay time data of the target real-time media resource data in the target data full link from a preset storage database, the target delay time data being collected and reported by each layer task node of the target data full link using a unified preset interface and stored in the preset storage database, the preset interface being configured using a software development kit; determining a task delay time of the target real-time media resource data based on each layer task node from the target delay time data, the task delay time being a time difference value between an actual time when the task node performs a data processing task and a reference time, the reference time being a time when a corresponding event of the target real-time media resource data is triggered; The second detection module is configured to determine a data quality detection result of the target real-time media resource data according to the task delay time of the target data full link, the data quality detection result being used to describe performance and availability of the target real-time media resource data.

11. An electronic device, comprising: The electronic device includes: one or more processors; a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the media resource data detection method of any one of claims 1-9.

12. A storage medium containing computer-executable instructions, wherein: The computer executable instructions, when executed by a computer processor, are configured to perform the media resource data detection method of any one of claims 1-9.

Images