A media frame anomaly detection method, device, equipment and medium

Abstract

The application relates to the technical field of video monitoring, and provides a media frame anomaly detection method, device, equipment and medium, which are used for reducing transmission overhead in a media frame anomaly detection process. The method is applied to a first process of a plurality of processes, the plurality of processes further include a second process, and the method comprises the following steps: receiving screening information sent by the second process, the screening information is file information of part of media files screened from a plurality of media files by the second process after collecting file information of the plurality of media files, the file information comprises a frame identifier, the frame identifier represents a serial number of a media frame of each media file, and whether the media frame corresponding to the frame identifier is abnormal is determined according to the screening information.

Description

Technical Field

[0001] This application relates to the field of video surveillance technology, specifically to a method, apparatus, device, and medium for detecting media frame anomalies. Background Technology

[0002] In the field of video surveillance technology, after video acquisition devices acquire video files and audio acquisition devices acquire audio files, detection devices can combine the video and audio files to obtain a single file with more complete information.

[0003] During the compositing process, video and audio files need to be transferred between multiple processes of the program and between multiple threads of each process. If a process or thread malfunctions, it will cause abnormal media frames, such as jitter or loss of media frames, which will lead to video stuttering.

[0004] Currently, detection equipment collects information from multiple locations across multiple processes. After summarizing the large amount of information collected, it determines whether there are any abnormal media frames. However, this method of summarizing a large amount of information places high demands on the processor performance of the detection equipment, resulting in significant overhead. Summary of the Invention

[0005] This application provides a media frame anomaly detection method, apparatus, device, and medium to reduce transmission overhead during the media frame anomaly detection process.

[0006] In a first aspect, embodiments of this application provide a media frame anomaly detection method, applied in a first process of multiple processes, wherein the multiple processes further include a second process, the method comprising:

[0007] The process receives filtering information sent by the second process, wherein the filtering information is the file information of a portion of the media files selected by the second process from the multiple media files after collecting file information of multiple media files, and the file information includes a frame identifier, wherein the frame identifier represents the sequence number of the media frame of each media file;

[0008] Based on the filtering information, determine whether the media frame corresponding to the frame identifier is abnormal.

[0009] In this embodiment, after collecting file information from multiple media files, the second process selects a portion of the file information from the multiple media files as filtering information and sends the filtering information to the first process. Since the filtering information received by the first process has been filtered by the second process, the transmission overhead between processes can be reduced, effectively reducing the performance consumption caused by information transmission between processes. Furthermore, the first process determines anomalies between two adjacent media frames based on the filtering information. Since the filtering information is relatively small, the processing efficiency of the first process can be improved, thereby improving the efficiency of media frame anomaly detection.

[0010] In one possible embodiment, the plurality of processes further includes a plurality of second processes, wherein the file information of the portion of the media files is collected at boundary points in the plurality of second processes;

[0011] Receive the filtering information sent by the second process, including:

[0012] The system receives filtering information and recording information sent by the plurality of second processes. The recording information includes a location identifier for collecting the file information, and the location identifier indicates the collection location point among the plurality of second processes when collecting the file information.

[0013] Based on the filtering information, determine whether the media frame corresponding to the frame identifier is abnormal, including:

[0014] Based on the filtering information, it is determined that the media frame corresponding to the frame identifier is abnormal;

[0015] Based on the location identifier, determine the second process among the plurality of second processes that caused the media frame anomaly corresponding to the frame identifier.

[0016] In this embodiment of the application, after the first process determines that the media frame corresponding to the frame identifier is abnormal, it can accurately determine the second process that caused the abnormality of the media frame from multiple second processes according to the location identifier, so as to facilitate finding the specific fault location point from the second process in order to restore the normal transmission of the media frame as soon as possible.

[0017] In one possible embodiment, the file information further includes a traceline identifier, which indicates the source of each media file;

[0018] Based on the filtering information, determine whether the media frame corresponding to the frame identifier is abnormal, including:

[0019] Based on the filtering information, it is determined that the media frame corresponding to the frame identifier is abnormal;

[0020] Based on the tracing line identifier, the source of the media file corresponding to the media frame of the frame identifier is determined.

[0021] In this embodiment of the application, after the first process determines that there is an anomaly in the media frame of a certain media file, it can determine the source of the media file according to the tracing line identifier, so as to determine the source of the media frame that caused the anomaly.

[0022] In one possible embodiment, the recorded information further includes the time of collection of the file information;

[0023] Based on the filtering information, determine whether the media frame corresponding to the frame identifier is abnormal, including:

[0024] For each media file in the aforementioned media files, two adjacent media frames are determined based on the frame identifier;

[0025] The presence of jitter between two adjacent media frames is determined based on whether the difference between the acquisition times of the two adjacent media frames meets a preset threshold condition.

[0026] In this embodiment of the application, the difference between the acquisition time of two adjacent media frames is used to determine whether there is jitter between the two adjacent media frames. This can further determine that the specific cause of the media frame abnormality is media frame jitter, which facilitates the subsequent recovery of the normal transmission of the media frame.

[0027] Secondly, embodiments of this application provide a media frame anomaly detection device, wherein the device is disposed in a first process of a plurality of processes, the plurality of processes further comprising a second process, and the device includes:

[0028] The receiving module is used to receive filtering information sent by the second process, wherein the filtering information is the file information of a portion of the media files selected by the second process from the multiple media files after collecting file information of multiple media files, and the file information includes a frame identifier, wherein the frame identifier represents the sequence number of the media frame of each media file;

[0029] The determination module is used to determine whether there is an anomaly in the media frame corresponding to the frame identifier based on the filtering information.

[0030] In one possible embodiment, the plurality of processes includes a plurality of second processes, and the file information of the portion of the media files is collected at boundary points in the plurality of second processes;

[0031] The receiving module is specifically used to receive filtering information and recording information sent by the plurality of second processes. The recording information includes a location identifier for collecting the file information, and the location identifier indicates the collection location point among the plurality of second processes when collecting the file information.

[0032] The determining module is specifically used to determine, based on the filtering information, that the media frame corresponding to the frame identifier is abnormal, and based on the location identifier, to determine the second process among the plurality of second processes that caused the abnormality of the media frame corresponding to the frame identifier.

[0033] In one possible embodiment, the file information further includes a traceline identifier, which indicates the source of each media file;

[0034] The determining module is specifically used to determine, based on the filtering information, that the media frame corresponding to the frame identifier is abnormal, and based on the tracing line identifier, to determine the source of the media file of the media frame corresponding to the frame identifier.

[0035] In one possible embodiment, the recorded information further includes the time of collection of the file information; the determining module is specifically used for:

[0036] For each media file in the aforementioned media files, two adjacent media frames are determined based on the frame identifier;

[0037] The presence of jitter between two adjacent media frames is determined based on whether the difference between the acquisition times of the two adjacent media frames meets a preset threshold condition.

[0038] Thirdly, embodiments of this application provide a media frame anomaly detection device, including:

[0039] At least one processor, and

[0040] A memory that is communicatively connected to the at least one processor;

[0041] The memory stores instructions executable by the at least one processor, which implements the method as described in any one of the first aspects by executing the instructions stored in the memory.

[0042] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the method as described in any one of the first aspects. Attached Figure Description

[0043] Figure 1 This application provides an illustration of an application scenario for a media frame anomaly detection method.

[0044] Figure 2 This application provides a schematic diagram of the interaction between multiple processes in an embodiment.

[0045] Figure 3This application provides a schematic diagram of the framework of a testing device.

[0046] Figure 4A A schematic diagram of a second process provided for embodiments of this application. Figure 1 ;

[0047] Figure 4B A schematic diagram of a second process provided for embodiments of this application. Figure 2 ;

[0048] Figure 4C A schematic diagram of a second process provided for embodiments of this application. Figure 3 ;

[0049] Figure 5 A schematic diagram of a first process provided for an embodiment of this application;

[0050] Figure 6 A structural diagram of a media frame anomaly detection device provided in an embodiment of this application;

[0051] Figure 7 This application provides a structural diagram of a media frame anomaly detection device. Detailed Implementation

[0052] To better understand the technical solutions provided in the embodiments of this application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0053] To reduce transmission overhead during media frame anomaly detection, this application provides a media frame anomaly detection method. This method can be executed by a media frame anomaly detection device; for simplicity, the media frame anomaly detection device will be referred to as the detection device below. The detection device can be implemented through a terminal or a server; the terminal can be, for example, a personal computer, and the server can be, for example, a physical server or a virtual server.

[0054] The following diagram illustrates the application scenario of this media frame anomaly detection method. Please refer to... Figure 1 This is a schematic diagram illustrating an application scenario of a media frame anomaly detection method provided in an embodiment of this application.

[0055] The application scenario includes: a first acquisition device 110, a second acquisition device 120, and a detection device 130. The first acquisition device 110 can be a device with video recording function, such as a video recorder, and the second acquisition device 120 can be a device with audio recording function, such as a audio recorder.

[0056] After the first acquisition device 110 acquires video, it sends it to the detection device 130. After the second acquisition device 120 acquires audio, it sends it to the detection device 130. After receiving the video and audio, the detection device 130 combines the video and audio to obtain a media file that contains both the video and the audio. During the combining process, it detects whether there are any abnormalities in the media frames. The combining process and the media frame abnormality detection process will be described in detail below.

[0057] It should be noted that, Figure 1 Taking a first acquisition device 110 and a second acquisition device 120 as an example, the number of first acquisition devices 110 and second acquisition devices 120 is not actually limited.

[0058] based on Figure 1 The application scenarios discussed below are as follows: Figure 1 This article will introduce the media frame anomaly detection method using a detection device as an example.

[0059] The detection device executes a media frame anomaly detection method by running a program. The program's execution involves interaction between multiple processes. Please refer to [link / reference needed]. Figure 2 This is a schematic diagram of the interaction between multiple processes provided in an embodiment of this application.

[0060] S201, The second process acquires multiple media files.

[0061] A process is a running program. The program corresponding to the media frame anomaly detection method involves the interaction between multiple processes. These multiple processes include secondary processes, which can be one or more; there is no limit to the number of secondary processes. When there are multiple secondary processes, each secondary process can acquire different media files. Media files can be video files or audio files.

[0062] The second process can acquire various media files, which will be introduced below.

[0063] In the first scenario, the first video, the first video stream, the second video, and the second video stream are treated as multiple media files.

[0064] Specifically, after the first acquisition device acquires the first video, it sends it to the detection device. The first video can be a video in a preset format, such as YUV format. When the detection device runs the program, the second process acquires the first video. The second process can be a video process (TaskVideo).

[0065] After the second process obtains the first video, it can encode the first video using a preset video encoding algorithm to obtain the first video bitstream (Video Bits, VBits). The preset video encoding algorithm is, for example, the H.264 video compression algorithm.

[0066] After the second process acquires the first video, it can also scale the first video, for example, by increasing or decreasing its resolution. Specifically, if the first video has a resolution of W x H, the obtained second video might have a resolution of W' x H', where W represents the number of pixels in the horizontal direction of the first video, H represents the number of pixels in the vertical direction of the first video, W' represents the number of pixels in the horizontal direction of the second video, and H' represents the number of pixels in the vertical direction of the second video. If the resolution of the first video is increased, then W' > W, H' > H. If the resolution of the first video is decreased, then W' > W. <W,H'<H。

[0067] After the second process obtains the second video, it can encode the second video using a preset video encoding algorithm to obtain the second video bitstream. Please refer to the content discussed above for the preset video encoding algorithm, which will not be repeated here.

[0068] Furthermore, the second process acquires the first video, the first video stream, the second video, and the second video stream as multiple media files.

[0069] The second scenario involves treating the audio and audio stream as multiple media files.

[0070] Specifically, the audio acquired by the second acquisition device is usually an analog signal. Therefore, after acquiring the audio, the second acquisition device can use Pulse Code Modulation (PCM) to convert the analog audio signal into a digital audio signal and send it to the detection device. When the detection device runs its program, the second process acquires the digital audio signal; this second process is, for example, the audio process (TaskAudio).

[0071] After the second process acquires the audio signal from the digital signal, it can further compress the audio by encoding it using a preset audio encoding algorithm to obtain the audio bitstream (Audio Bits, ABits). A preset audio encoding algorithm could be, for example, Sub-Band Code (SBC).

[0072] Furthermore, the second process acquires the audio and audio stream as multiple media files.

[0073] In the third scenario, the first video stream, the second video stream, the audio stream, the main stream, and the sub-streams are treated as multiple media files.

[0074] Specifically, when the detection device is running its program, the second process can acquire the first video stream, the second video stream, and the audio stream sent by other processes. For example, TaskVideo sends the first and second video streams, and TaskAudio sends the audio stream. In this case, the second process is, for example, a stream mixing process (TaskStreamMux). After acquiring the first video stream, the second video stream, and the audio stream, the second process can package the first video stream and the audio stream to obtain the main stream (StreamMain), and package the second video stream and the audio stream to obtain the sub-stream (StreamSub).

[0075] Furthermore, the second process treats the first video stream, the second video stream, the audio stream, the main stream, and the sub-stream as multiple media files.

[0076] To more clearly illustrate the process of the second process acquiring multiple media files, the following will combine... Figure 3 The testing equipment shown will be explained. Please refer to... Figure 3 This application provides a schematic diagram of the framework of a detection device. The detection device includes: a video acquisition module 301, a video scaling module 302, a first video encoding module 303, a second video encoding module 304, an audio acquisition module 305, an audio encoding module 306, a main stream mixing module 307, and a sub-stream mixing module 308. The function of each module is described in detail below.

[0077] The video acquisition module 301 is used to acquire a first video. The video scaling module 302 is used to scale the first video to obtain a second video. The first video encoding module 303 is used to encode the first video to obtain a first video bitstream. The second video encoding module 304 is used to encode the second video to obtain a second video bitstream. The audio acquisition module 305 is used to acquire audio. The audio encoding module 306 is used to encode the audio to obtain an audio bitstream. The main bitstream mixing module 307 is used to package the first video bitstream and the audio bitstream into a main bitstream. The sub-bitstream mixing module 308 is used to package the second video bitstream and the audio bitstream into a sub-bitstream.

[0078] S202, The second process collects file information.

[0079] After the second process acquires multiple media files, it can collect file information for each media file. This file information includes at least a Sequence Identity Document (SID), and may also include a Line Identity Document (LID). The frame identifier and the line identifier are described below.

[0080] 1. Frame identifier.

[0081] Frame identifiers are used to represent the sequence number of media frames in each media file. Each media file includes multiple media frames, such as multiple video frames and multiple audio frames. Frame identifiers include, for example, SID = $Sn and SID = $Sm.

[0082] 2. Tracking line markings.

[0083] The tracing line identifier indicates the source of each media file, including the first video, the second video, audio, the main stream, and the sub-stream. For example, LID=$Vn indicates that the media file originates from the first video, and LID=$V'n indicates that the media file originates from a scaled-down version of the first video, i.e., from the second video.

[0084] As one embodiment, the second process can also save record information when collecting file information from each of the multiple media files. The meaning of the file information is explained above and will not be repeated here. The record information includes at least one of the following: the collection time of the collected file information and a Point Identity Document (PID). The PID indicates the collection location point among the multiple second processes when collecting the file information. There are various methods for representing the PID, which will be illustrated with examples below.

[0085] (1) Numerical number.

[0086] Location identifiers can be represented by numerical codes, such as "1", "2", etc.

[0087] (2) Instruction memory address and process identifier.

[0088] Location identifiers can also be represented by instruction memory addresses and process identifiers. The instruction is the collection instruction for collecting file information, the instruction memory address represents the memory address where the collection instruction is executed, and the process identifier is used to uniquely identify each process.

[0089] S203, The second process obtains the filtering information.

[0090] To reduce transmission overhead, in this embodiment of the application, the second process can filter the file information of multiple media files to obtain filtering information, wherein the filtering information is the file information of some media files among the multiple media files.

[0091] This involves how the second process filters file information from multiple media files. There are several ways for the second process to filter file information, which will be introduced below.

[0092] The first method allows for arbitrary filtering of the second process.

[0093] The second process can arbitrarily filter some media files from multiple media files, using the file information of some media files as the filtering information.

[0094] The second method involves filtering based on location identifiers.

[0095] The second process can determine the file information of some media files collected at the boundary points as filtering information based on the location identifier. This filtering information can be called boundary information.

[0096] For example, location identifiers are represented by numerical numbers. The closer a sampling point in the second process is to a boundary point, the larger the value of the location identifier corresponding to that sampling point. From multiple location identifiers saved in a certain second process, the sampling point corresponding to the location identifier with the largest value is determined as the boundary point in that second process, and the file information corresponding to the location identifier with the largest value is determined as the filtering information. Specifically, when the second process samples audio file information, the location identifier corresponding to its sampling point is 1. When the second process samples audio stream file information, the corresponding location identifier is 2, indicating that the sampling point corresponding to location identifier 2 is a boundary point. The file information corresponding to location identifier 2, i.e., the audio stream file information, is determined as the filtering information.

[0097] Alternatively, location identifiers can be represented by instruction memory addresses and process identifiers. From multiple location identifiers saved by a second process, if the instruction memory address of a certain location identifier is a preset instruction memory address, then the collection location point corresponding to that location identifier is determined as the boundary point of the second process, and the file information corresponding to that location identifier is determined as the filtering information.

[0098] S204, The second process sends filtering information to the first process.

[0099] After obtaining the filtering information, the second process can either send the filtering information directly to the first process, or send the filtering information and the recording information together to the first process. Here, the first process refers to all processes other than the second process; the second process can be considered the upstream of the first process, and the first process can be considered the downstream of the second process.

[0100] To more clearly illustrate the processes in S203 and S204, the following will combine... Figures 4A-4C The second process will be described below. Please refer to... Figure 4A This application provides a schematic framework for a second process. Figure 1The second process, TaskVideo, includes an acquisition module 401 and a Domain Occur (DMO) module 402. The acquisition module 401 acquires multiple media files of video type, including a first video, a first video stream, a second video, and a second video stream. The DMO module 402 collects file information for each media file and saves recording information, sending the filtering and recording information to the first process. The information within the dashed box indicated by the dashed arrow between the acquisition module 401 and the DMO module 402 is the file information and recording information for each media file. For example, the first video's LID = $Vn, PID = 1, SID = $Sn; the second video's LID = $V'n, PID = 1, SID = $Sn; the first video stream's LID = $Vn, PID = 2, SID = $Sn; and the second video stream's LID = $V'n, PID = 2, SID = $Sn.

[0101] DMO module 402 includes a First-Tier Algorithm (FTA) module 403 and a Service Publisher (SrvPub) module 404. The FTA module 403 collects file information for each media file, saves recording information, and determines filtering information. The SrvPub module 404 sends the filtering and recording information to the first process. The information within the dashed box between the FTA module 403 and the SrvPub module 404 is the filtering and recording information, such as LID = $Vn, PID = 2, SID = $Sn for the first video stream and LID = $V'n, PID = 2, SID = $Sn for the second video stream.

[0102] Please refer to Figure 4B This application provides a schematic framework for a second process. Figure 2 The second process, TaskAudio, includes an acquisition module 401 and a DMO module 402. The acquisition module 401 acquires audio media files, including audio and audio streams. The DMO module 402 collects file information for each media file and saves recording information, then sends the filtering and recording information to the first process. The information within the dashed box indicated by the dashed arrow between the acquisition module 401 and the DMO module 402 is the file information and recording information for each media file, such as LID=$An, PID=1, SID=$Sm for audio and LID=$An, PID=2, SID=$Sm for audio streams.

[0103] DMO module 402 includes FTA module 403 and SrvPub module 404. The functions of FTA modules 403 and 404 are discussed previously and will not be repeated here. The information within the dashed box between FTA module 403 and SrvPub module 404 is filtering and recording information, such as the audio stream's LID = $An, PID = 2, and SID = $Sm.

[0104] Please refer to Figure 4C This application provides a schematic framework for a second process. Figure 3 The second process, TaskStreamMux, includes an acquisition module 401 and a DMO module 402. The acquisition module 401 is used to acquire media files of video and audio types, including a first video stream, a second video stream, an audio stream, a main stream, and sub-streams. The DMO module 402 is used to collect file information for each media file and save the recording information, and send the filtering information and recording information to the first process.

[0105] The information within the dashed box indicated by the dashed arrow between module 401 and DMO module 402 is the file information and record information of each media file. For example, the LID of the first video stream is $Vn, PID is 3, and SID is $Sn; the LID of the second video stream is $V'n, PID is 3, and SID is $Sn; the LID of the audio stream is $An, PID is 4, and SID is $Sm; the LID of the main stream is $Vn+$An, PID is 5, and SID is $Sn+$Sm; and the LID of the sub-stream is $V'n+$An, PID is 5, and SID is $Sn+$Sm.

[0106] DMO module 402 includes FTA module 403 and SrvPub module 404. The functions of FTA modules 403 and 404 are discussed previously and will not be repeated here. The information within the dashed box between FTA module 403 and SrvPub module 404 is filtering and recording information. For example, the main stream's LID = $Vn + $An, PID = 5, SID = $Sn + $Sm, and the sub-stream's LID = $V'n + $An, PID = 5, SID = $Sn + $Sm.

[0107] S205, The first process receives the filtering information.

[0108] If the second process sends filtering information, the first process receives filtering information; if the second process sends both filtering and recording information, the first process receives both filtering and recording information.

[0109] S206. The first process checks whether there are any abnormalities in the media frames.

[0110] If the first process only receives filtering information, it can determine whether media frames are lost based on whether the frame identifiers of the same media file are consecutive.

[0111] Specifically, for each media file in a subset of media files, the first process can determine whether the frame identifiers of the same media file are consecutive based on the frame identifiers in the filtering information. If the frame identifiers of the same media file are not consecutive, it is determined that some media frames of that media file are missing; if the frame identifiers of the same media file are consecutive, it is determined that no media frames of that media file are missing. For example, if the frame identifiers of the same media file are $S1, $S2, $S4, $S5, and $S6, it means that the media frame with frame identifier $S3 is missing.

[0112] As one embodiment, if the first process receives filtering information and recording information, and the recording information includes the acquisition time, the first process can determine whether there is jitter between two adjacent media frames based on whether the difference between the acquisition times of two adjacent media frames meets a preset threshold condition.

[0113] Specifically, for each media file in a subset of media files, the first process can determine two adjacent media frames based on the frame identifiers in the filtering information. For example, if the frame identifiers of two media frames are SID=$S1 and SID=$S2, then the two media frames are determined to be adjacent. The acquisition time of each of the two adjacent media frames is determined based on the recorded information. If the difference between the acquisition times of the two adjacent media frames does not meet a preset threshold condition, then jitter is determined to exist between the two adjacent media frames. If the difference between the acquisition times of the two adjacent media frames meets the preset threshold condition, then jitter is determined to be absent between the two adjacent media frames.

[0114] The preset threshold condition can be that the difference between the acquisition times of two adjacent media frames is less than a preset threshold, for example, a preset threshold of 40ms. If the difference is greater than 40ms, it is determined that there is jitter between the two adjacent media frames; if the difference is less than 40ms, it is determined that there is no jitter between the two adjacent media frames. Alternatively, the preset threshold condition can be that the difference between the acquisition times of two adjacent media frames is within a preset threshold range, for example, a preset threshold range of [30ms, 40ms]. If the difference is 45ms, which is not within the preset threshold range, it is determined that there is jitter between the two adjacent media frames. If the difference is 35ms, which is within the preset threshold range, it is determined that there is no jitter between the two adjacent media frames.

[0115] It should be noted that if the first process receives filtering information and recording information, the first process can also determine whether media frames are lost based on whether the frame identifiers of the same media file are consecutive. For details on how to determine whether media frames are lost, please refer to the previous discussion, which will not be repeated here.

[0116] To more clearly illustrate the processes in S205 and S206, the following will be combined with... Figure 5 The first process shown will be introduced below. Please refer to... Figure 5 This application provides a schematic diagram of the framework of a first process. The first process is a main process (TaskMain) and includes a Domain Process (DMP) module 501. The DMP module 501 is used to receive filtering information and recording information sent by the SrvSub module of the second process, and to detect whether there is jitter in the media frame. Figure 5 Taking TaskVideo, TaskStreamMux, and TaskAudio as examples, there is actually no limit to the number of second processes.

[0117] DMP module 501 includes a Service Subscriber (SrvSub) module 502 and a Second-Tier Algorithm (STA) module 503. The SrvSub module 502 receives filtering and recording information sent by the SrvSub modules of the second process. For example, the filtering and recording information sent by the TaskVideo SrvSub module is: LID = $Vn, PointID = 2, SID = $Sn and LID = $V'n, PointID = 2, SID = $Sn. The filtering and recording information sent by the TaskStreamMux SrvSub module is: LID = $Vn + $An, PID = 5, SID = $Sn + $Sm and LID = $V'n + $An, PID = 5, SID = $Sn + $Sm. The filtering and recording information sent by the TaskAudio SrvSub module is: LID = $An, PointID = 2, SID = $Sm. The STA module 503 is used to detect whether there are any abnormalities in the media frames based on the filtering information.

[0118] As an example, the FTA module of the second process can also perform media frame anomaly detection based on the file information and record information of multiple media files. The media frame anomaly detection process is described in the previous text and will not be repeated here.

[0119] S207. The first process identifies the second process among multiple second processes that caused the media frame anomaly.

[0120] After the first process determines that the media frame is abnormal, it can further determine the second process that caused the media frame abnormality among multiple second processes based on the position identifier in the recorded information.

[0121] Because the location identifiers are represented differently, the first process determines the second process in different ways, which will be described below.

[0122] In the first scenario, the location identifier is represented by the instruction memory address and the process identifier.

[0123] If the location identifier is represented by the instruction memory address and the process identifier, then the first process can directly determine the second process among multiple second processes that caused the media frame abnormality based on the process identifier.

[0124] For example, if the process identifier is "TaskVideo", then among multiple second processes, the second process that causes media frame abnormalities is the video process TaskVideo.

[0125] In the second scenario, the location identifier is represented by a numerical code.

[0126] If the position identifiers are represented by numerical numbers, the first process can determine the second process causing the media frame anomaly among multiple second processes based on the position identifiers and a pre-stored correspondence. The pre-stored correspondence includes multiple position identifiers and the second process corresponding to each position identifier. For example, position identifier 2 corresponds to the second process "TaskVideo", position identifier 3 corresponds to the second process "TaskAudio", and position identifier 5 corresponds to the second process "TaskStreamMux".

[0127] S208. The first process determines the source of the media file corresponding to the abnormal media frame.

[0128] After the first process determines that the media frame corresponding to the frame identifier is abnormal, it can also determine the source of the media file corresponding to that frame identifier based on the tracing line identifier in the filtering information. For example, LID = $Vn indicates that the media file comes from the first video, LID = $V'n indicates that the media file comes from the scaled first video, that is, from the second video, LID = $An indicates that the media file comes from the audio, LID = $Vn + $An indicates that the media file comes from the first video and audio, that is, from the main stream, and LID = $V'n + $An indicates that the media file comes from the second video and audio, that is, from the sub-stream.

[0129] It should be noted that the execution order of S207 and S208 is arbitrary and optional. If the first process only receives the filtering information, then S207 and S208 will not be executed. If the recorded information does not include the position identifier, then S207 will not be executed. If the filtering information does not include the tracking line identifier, then S208 will not be executed.

[0130] Based on the same inventive concept, this application provides a media frame anomaly detection device, which is essentially set in the first process described above. Please refer to... Figure 6 The device includes:

[0131] The receiving module 601 is used to receive filtering information sent by the second process. The filtering information is the file information of a portion of the media files selected by the second process from the multiple media files after collecting the file information of multiple media files. The file information includes a frame identifier, which represents the sequence number of the media frame of each media file.

[0132] The determination module 602 is used to determine whether there is an anomaly in the media frame corresponding to the frame identifier based on the filtering information.

[0133] In one possible embodiment, the multiple processes include multiple second processes, and the file information of some media files is collected at boundary points in the multiple second processes;

[0134] The receiving module 601 is specifically used to receive filtering information and recording information sent by multiple second processes. The recording information includes the location identifier of the collected file information, and the location identifier indicates the boundary point among the multiple second processes when collecting file information.

[0135] The determination module 602 is specifically used to determine, based on the filtering information, that the media frame corresponding to the frame identifier is abnormal, and based on the position identifier, to determine the second process among multiple second processes that caused the media frame corresponding to the frame identifier to be abnormal.

[0136] In one possible embodiment, the file information also includes a traceline identifier, which indicates the source of each media file;

[0137] The determination module 602 is specifically used to determine, based on the filtering information, that the media frame corresponding to the frame identifier is abnormal, and based on the tracing line identifier, to determine the source of the media file of the media frame corresponding to the frame identifier.

[0138] In one possible embodiment, the recorded information also includes the acquisition time of the acquired file information; the determination module 602 is specifically used for:

[0139] For each media file in a subset of media files, determine the two adjacent media frames based on the frame identifier;

[0140] The presence of jitter between two adjacent media frames is determined by whether the difference between the acquisition times of two adjacent media frames meets a preset threshold condition.

[0141] As one example, Figure 6 The media frame anomaly detection device described above can implement any of the media frame anomaly detection methods discussed above, and will not be repeated here.

[0142] Based on the same inventive concept, embodiments of this application provide a media frame anomaly detection device, please refer to... Figure 7 The device includes:

[0143] At least one processor 701, and

[0144] A memory 702 communicatively connected to at least one processor 701;

[0145] The memory 702 stores instructions that can be executed by at least one processor 701. The at least one processor 701 implements any of the media frame anomaly detection methods discussed above by executing the instructions stored in the memory 702.

[0146] Processor 701 can be a central processing unit (CPU), a digital processing unit, or a graphics processor, or a combination thereof. Memory 702 can be volatile memory, such as random-access memory (RAM); memory 702 can also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 702 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures, accessible by a computer, but is not limited thereto. Memory 702 can be a combination of the above-mentioned memories.

[0147] As one example, Figure 7 The processor 701 can implement any of the media frame anomaly detection methods discussed above. The processor 701 can also implement the methods discussed above. Figure 6 The functions of the media frame anomaly detection device are discussed.

[0148] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the media frame anomaly detection method as described above.

[0149] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0150] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0151] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0152] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0153] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for detecting media frame anomalies, characterized in that, The method, applied to a first process comprising multiple processes, wherein the multiple processes further include multiple second processes, includes: The system receives filtering information and recording information sent by the plurality of second processes. The filtering information consists of file information from a subset of media files selected by the second process after collecting file information from multiple media files. The file information includes a frame identifier, which represents the sequence number of a media frame in each media file. The file information is collected at boundary points within the plurality of second processes. The recording information includes a location identifier for collecting the file information, which represents the collection location point within the plurality of second processes when collecting the file information. Among the multiple location identifiers stored by each second process, the location point corresponding to the location identifier with the largest value is the boundary point within that second process. Based on the filtering information, it is determined that the media frame corresponding to the frame identifier is abnormal; Based on the location identifier, determine the second process among the plurality of second processes that caused the media frame anomaly corresponding to the frame identifier.

2. The method as described in claim 1, characterized in that, The file information also includes a tracing line identifier, which indicates the source of each media file; after determining that the media frame corresponding to the frame identifier is abnormal based on the filtering information, it further includes: Based on the tracing line identifier, the source of the media file corresponding to the media frame of the frame identifier is determined.

3. The method as described in claim 1, characterized in that, The recorded information also includes the time when the file information was collected; Based on the filtering information, it is determined that the media frame corresponding to the frame identifier is abnormal, including: For each media file in the aforementioned media files, two adjacent media frames are determined based on the frame identifier; If the difference between the acquisition times of two adjacent media frames meets a preset threshold condition, it is determined that there is jitter between the two adjacent media frames.

4. A media frame anomaly detection device, characterized in that, The device is disposed in a first process of a plurality of processes, the plurality of processes further including a plurality of second processes, the device comprising: A receiving module is configured to receive filtering information and recording information sent by the plurality of second processes; wherein, the filtering information is the file information of a portion of the media files selected by the second process after collecting file information from the plurality of media files, the file information including a frame identifier, the frame identifier representing the sequence number of the media frame of each media file; the file information is collected at boundary points in the plurality of second processes; the recording information includes a location identifier for collecting the file information, the location identifier representing the collection location point in the plurality of second processes when collecting the file information; among the multiple location identifiers stored by each second process, the location point corresponding to the location identifier with the largest value is the boundary point in that second process; The determination module is used to determine, based on the filtering information, that the media frame corresponding to the frame identifier is abnormal, and based on the location identifier, to determine the second process among the plurality of second processes that caused the abnormality of the media frame corresponding to the frame identifier.

5. The apparatus as described in claim 4, characterized in that, The file information also includes a traceline identifier, which indicates the source of each media file; The determining module is further configured to, after determining that the media frame corresponding to the frame identifier is abnormal based on the filtering information, determine the source of the media file of the media frame corresponding to the frame identifier based on the tracing line identifier.

6. The apparatus as claimed in claim 4, characterized in that, The recorded information also includes the time of collection of the file information; the determining module is specifically used for: For each media file in the aforementioned media files, two adjacent media frames are determined based on the frame identifier; If the difference between the acquisition times of two adjacent media frames meets a preset threshold condition, it is determined that there is jitter between the two adjacent media frames.

7. A media frame anomaly detection device, characterized in that, include: At least one processor, and A memory that is communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the method as described in any one of claims 1-3 by executing the instructions stored in the memory.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1-3.

Images