Implementation method and system of video image test element based on Gstreamer multimedia framework

Abstract

The present invention discloses an implementation method and system of a video image test element based on a Gstreamer multimedia framework. In the plug-in loading stage of the Gstreamer multimedia framework, a video image test element is registered in the Gstreamer multimedia framework, an upstream element pushes data buffers filled with video image data into the video image test element one by one, and an image AI test module of the video image test element performs data calculation, and the image AI test module obtains image histogram distribution values ​​of each data buffer according to the calculation and then compares them, thereby realizing automatic test verification of video image data, and improving the accuracy, reproducibility and test efficiency of video image testing.

Description

Technical Field

[0001] The invention relates to a method and a system for realizing a video image test element based on a Gstreamer multimedia framework. Background Art

[0002] In recent years, with the continuous innovation and iteration of technologies such as 5G, virtual reality, and artificial intelligence, media communication has continued to move towards diversification, high-end, and intelligence.

[0003] In this context, the demand for video content production and consumption has increased dramatically. As a highly flexible open source multimedia framework, the Gstreamer multimedia framework has been widely used in many fields due to its powerful functions and modular design, including streaming media, video editing, audio processing, and various embedded systems. It has also been adopted by many small businesses and developers to build customized multimedia solutions.

[0004] The Gstreamer multimedia framework implements a logical monitoring tool (GstValidate) internally, which allows the system to check whether the elements in the pipeline comply with some rules that the Gstreamer multimedia framework must follow so that they can interact correctly.

[0005] The logic monitoring tool (GstValidate) automatically runs a series of test cases to detect whether the element meets the expected behavior and performance indicators, and allows users to obtain detailed information about what the element did not handle correctly, and finally generates a test report.

[0006] At this stage, testers need to test whether the video images based on the Gstreamer multimedia framework are processed normally. The operation steps are as follows:

[0007] S1. Testers need to write a validation test file (.validatetest) for validating the flow. In the validation test file, the element to be monitored and the pad of the element need to be specified, as well as the data buffer and event flowing through the pad. Testers can also specify the operation scenarios to be run to find and change the status of the test pipeline.

[0008] S2. The validation test file (.validatetest) written by the tester is registered in the Gstreamer multimedia framework through the logic monitoring tool (GstValidate), and a new test unit (testsuites) is generated on the test list;

[0009] S3. When the test suite is run for the first time, the data buffer and events of the specified pad previously subscribed in the validation test file (.validatetest) will be recorded in the log file, which will be used as the expected log file;

[0010] S4. Subsequent re-execution of the test suite will also record the generated data buffer and events, and compare the data buffer and events with the expected log file. Any difference between the two will be reported as a test failure.

[0011] S5. The tester can re-determine a new expected log file by deleting the expected log file and running the test suite again.

[0012] The use of the logic monitoring tool (GstValidate) to write integrated test units (testsuites) focuses on testing at the high-level framework layer. It is not suitable for refined data flow test verification. For typical video playback screen distortion, video jump image not updated, or abnormal image color after video jump, the test method mainly relies on manual execution and visual observation. The test efficiency is low and the test results are limited by the subjective judgment of the tester, lacking test accuracy and reproducibility. Summary of the invention

[0013] The purpose of the present invention is to provide a method and system for realizing a video image test element based on a Gstreamer multimedia framework in view of the deficiencies in the prior art, which can realize automatic verification of video data, improve the accuracy and reproducibility of video image testing, and enhance the testing efficiency of video images.

[0014] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0015] Implementation method of video image test component based on Gstreamer multimedia framework,

[0016] Includes steps:

[0017] Step 1. In the plug-in loading stage of the Gstreamer multimedia framework, a video image test component is registered in the Gstreamer multimedia framework, so that the source pad of the component that needs to test the output video data is connected to the video image test component. The video image test component will create a queue and a thread when registering;

[0018] Step 2. The upstream component pushes the events, queries, and data buffers filled with video image data one by one into the video image test component;

[0019] Step 3. The data buffer of the video image test component must first enter the queue;

[0020] Enter the event and query of the video image test component. The video image test component needs to determine whether it needs to enter the queue according to the type of event and query;

[0021] Step 4. The thread dequeues the data buffers, events, and queries stored in the queue one by one;

[0022] The thread submits the dequeued data buffer to the image AI test module in the video image test component for data calculation, and obtains the test result based on the image histogram distribution numerical value obtained by the data calculation. The obtained test result will be saved to the output file directory, and then the thread exits; after the thread pushes the dequeued events and queries to the downstream components, the thread exits.

[0023] By adopting the above technical solution, the upstream component pushes the data buffers filled with video image data into the video image test component one by one, and the image AI test module of the video image test component performs data calculation. The image AI test module obtains the image histogram distribution values ​​of each data buffer according to the calculation and compares them, thereby realizing automatic test verification of video image data, improving the accuracy, reproducibility and efficiency of video image testing; since the video image test component is registered in the plug-in loading stage of the Gstreamer multimedia framework, it exists in the Gstreamer multimedia framework in the form of a plug-in, so the user can arbitrarily select different plug-ins according to test needs to build a complex multimedia test pipeline, which is simpler and easier to use; and since the video image test component is in the form of a plug-in, it will not make any modifications to the input video data, which can better ensure the compatibility of the test scene; after the test results obtained by the image AI test module are saved in the output file directory, the test results will exist as a reference file, and the test results of the next round of data buffers will be associated with the reference file for further similarity comparison, thereby further improving the accuracy of the video image test.

[0024] Furthermore, before the data buffer entering the video image test component in Step 3 enters the queue, it is necessary to first acquire a mutex lock. After the data buffer successfully acquires the mutex lock, the state of the queue is queried, including the following situations:

[0025] A. If the query result shows that the queue is not full, the data buffer can enter the queue, and then send a wake-up signal to the thread, waiting to be dequeued by the thread;

[0026] B. If the query result shows that the queue is full, the data buffer needs to release the mutex lock and go into sleep mode. After the data buffer is awakened, it needs to reacquire the mutex lock.

[0027] Furthermore, in Step 3, the video image test component needs to determine whether it needs to enter the queue based on the type of event and query, including the following situations:

[0028] A. If the event entering the video image test component is a serialized event, the video image test component determines that the event needs to enter the queue;

[0029] B. If the event entering the video image test component is a non-serialized event, the video image test component needs to process the event immediately;

[0030] C. If the query entering the video image test component is a serialized query, the video image test component determines that the query needs to enter the queue;

[0031] D. If the query entering the video image test component is a non-serialized query, the video image test component needs to process the query immediately.

[0032] Furthermore, after the image AI test module in Step 4 starts to receive the data buffer pushed by the thread, it is necessary to first determine the current test mode.

[0033] Furthermore, if the image AI test module determines that the current test mode is in the full test mode, the test steps for the data buffer are as follows:

[0034] Step 4A.1. After receiving the data buffer pushed by the thread, the similarity comparison submodule calculates the histogram distribution value of the current data buffer and caches the histogram distribution value result of the current data buffer;

[0035] Step 4A.2. Determine whether the current data buffer is the first frame data buffer;

[0036] Step 4A.2.1. If the current data buffer is the first frame data buffer, directly determine that the current data buffer is normal and exit from the thread;

[0037] Step 4A.2.2. If the current data buffer is not the first frame data buffer, compare the histogram distribution value of the current data buffer with the histogram distribution value of the previous data buffer;

[0038] Step 4A.3. After the similarity comparison submodule obtains the comparison result of the histogram distribution value of this round according to Step 4A.2.2, it pushes the comparison result to the report processing submodule;

[0039] Step 4A.4. The report processing submodule saves the comparison results of the received histogram distribution values ​​in this round to the output file directory as a reference file for the next round of testing, and determines whether the current data buffer has passed the test based on whether the comparison results of the histogram distribution values ​​in this round exceed the threshold.

[0040] Furthermore, in Step 4A.4, if the comparison result of the histogram distribution value in this round exceeds the threshold, the report processing submodule will submit an error message to the pipeline bus, determine that the current data buffer fails the test and cause it to exit from the thread.

[0041] Furthermore, in Step 4A.4, if the comparison result of the histogram distribution value in this round is that it does not exceed the threshold, the report processing submodule will query the output file directory whether there is a reference file, including the following situations:

[0042] A. If the report processing submodule does not find the reference file in the output file directory, the report processing submodule determines that the current data buffer has passed the test and causes it to exit from the thread;

[0043] B. If the report processing submodule finds that there is a reference file in the output file directory, the report processing submodule will read the reference file and retrieve the comparison data based on the timestamp of the data buffer in the reference file. It will compare the retrieved comparison data with the comparison result of the current round of histogram distribution values ​​to determine whether the current data buffer has passed the test.

[0044] Furthermore, if the comparison result obtained by comparing the retrieved comparison data with the comparison result of the current round of histogram distribution values ​​exceeds the threshold, the report processing submodule will submit an error message to the pipeline bus, determine that the current data buffer fails the test and cause it to exit from the thread.

[0045] Furthermore, if the comparison result obtained by comparing the retrieved comparison data with the comparison result of the current round of histogram distribution values ​​is not above the threshold, the report processing submodule determines that the current data buffer has passed the test and causes it to exit from the thread.

[0046] Furthermore, if the image AI test module determines that the current test mode is in the skip test mode, the test steps for the data buffer are as follows:

[0047] Step 4B.1. Cache at least one data buffer in the queue;

[0048] Step 4B.2. If a skipping event is detected, the image AI test module continues to monitor several data buffers;

[0049] Step 4B.3. Refresh the queue in the callback function of the jump event, drain the content in the queue, and submit the last data buffer pushed into the queue to the similarity comparison submodule;

[0050] Step 4B.4. The similarity comparison submodule calculates the histogram distribution value of the data buffer and caches the histogram distribution value result of the data buffer;

[0051] Step 4B.5. After the jump event ends, the Gstreamer multimedia framework relocates to the jump position to obtain the source content data, and continues to push new data buffers to the video image test component;

[0052] Step 4B.6. The similarity comparison submodule receives new data buffers one by one, calculates the histogram distribution value thereof and compares it with the histogram distribution value of the previous data buffer to obtain the comparison result;

[0053] Step 4B.7. Push the comparison results one by one to the report processing submodule, and the report processing submodule saves the comparison results to the output file directory; the report processing submodule determines whether the video image after the jump is abnormal based on whether the comparison results of the evaluation histogram distribution values ​​conform to the rules.

[0054] Furthermore, in the skip test mode, if no skip event is detected, it is necessary to determine the number of data buffers that the image AI test module continuously monitors.

[0055] Further, if the number of data buffers continuously monitored by the image AI test module is zero, it is determined that the data buffer test in the queue has passed.

[0056] Furthermore, if the number of data buffers continuously monitored by the image AI test module is not zero, the test steps are as follows:

[0057] Step4C.1. The data buffers enter the similarity comparison submodule one by one;

[0058] Step 4C.2. The similarity comparison submodule calculates the histogram distribution value of the data buffer and caches the histogram distribution value result of the current data buffer;

[0059] Step 4C.3. Compare the histogram distribution value of the current data buffer with the histogram distribution value of the previous data buffer, and then send the comparison result to the report processing submodule;

[0060] Step 4C.4. The report processing submodule saves the comparison result of the received histogram distribution value to the output file directory and decrements the value of the monitored data buffer;

[0061] Step 4C.5. The report processing submodule determines whether an abnormality occurs in the current data buffer based on whether the comparison result of the histogram distribution values ​​conforms to the rules.

[0062] Furthermore, in Step 4C.5, if the report processing submodule determines that the comparison result of the histogram distribution value does not conform to the regularity, an error message will be submitted to the pipeline bus, and it will be determined that the current data buffer is abnormal.

[0063] Furthermore, in Step 4C.5, if the report processing submodule determines that the comparison result of the histogram distribution value conforms to the rule, the report processing submodule will query the output file directory whether there is a reference file, including the following situations:

[0064] A. If the report processing submodule does not find the reference file in the output file directory, the report processing submodule determines that the current data buffer is tested normally;

[0065] B. If the report processing submodule finds that there is a reference file in the output file directory, the report processing submodule will read the reference file and retrieve the comparison data based on the timestamp of the data buffer in the reference file. It will compare the retrieved comparison data with the comparison result of the current round of histogram distribution values ​​to determine whether the current data buffer has passed the test.

[0066] Furthermore, if the comparison result obtained by comparing the retrieved comparison data with the comparison result of the histogram distribution value is inconsistent with the law, the report processing submodule will submit an error message to the pipeline bus and determine that the current data buffer is a test exception.

[0067] Further, if the comparison result obtained by comparing the retrieved comparison data with the comparison result of the histogram distribution value is consistent with the law, the report processing submodule determines that the current data buffer is tested normally.

[0068] A system for video image testing components based on the Gstreamer multimedia framework.

[0069] include

[0070] Components, which are the basic building blocks of pipelines, are used to interact with the outside world through pads and represent their own characteristics or functions to other components in the pipeline;

[0071] Pads are used to connect components to the outside world and describe the media types that components can handle;

[0072] Pipeline: A pipeline is a streaming media processing pipeline composed of pads of different components connected in sequence, which is used to enable data to be processed by each component while flowing through the pipeline;

[0073] The components include a video image test component, which exists in the Gstreamer multimedia framework in the form of a plug-in. The video image test component can be connected to the source pad of the component that needs to test the output video data, and then receive the data buffer filled with video image data pushed by the upstream component, and test the data buffer to determine whether the video image in the data buffer is normal.

[0074] Furthermore, the video image test component includes a queue and a thread. The queue and the thread exist in the video image test component. The queue is used to store data buffers, serialized events, and serialized queries pushed down by upstream components; the thread is used to dequeue the contents of the queue one by one.

[0075] Furthermore, the video image test element includes an image AI test module, which exists in the video image test element. The image AI test module is used to perform data calculations and determine whether the video image in the data buffer is normal based on the image histogram distribution value obtained by the calculation.

[0076] Furthermore, the image AI test module includes a similarity comparison submodule and a report processing submodule. The similarity comparison submodule is used to operate on the received data buffer to obtain its histogram distribution value. At the same time, the similarity comparison submodule can also compare the histogram distribution value of the current data buffer with the histogram distribution value of the previous data buffer, and push the comparison result to the report processing submodule; the report processing submodule can save the comparison result as a reference file, and match the current comparison result with the reference file of the previous round to obtain the test result.

[0077] The present invention has the following beneficial effects due to the adoption of the above technical solution:

[0078] 1. The present invention uses the image AI test module to obtain the image histogram distribution values ​​of each data buffer according to calculations and then compares them, thereby realizing automatic testing of video image data and improving the accuracy, reproducibility and test efficiency of video image testing;

[0079] 2. The video image test component of the present invention exists in the form of a plug-in in the Gstreamer multimedia framework, so users can arbitrarily select different plug-ins to build a complex multimedia test pipeline according to test needs, which is simpler and easier to use;

[0080] 3. The video image test component of the present invention exists in the Gstreamer multimedia framework in the form of a plug-in, which does not make any modification to the input video data, and can better ensure the compatibility of the test scene;

[0081] 4. After the test results obtained by the image AI test module of the present invention are saved in the output file directory, the test results will exist as a reference file, and the test results of the next round of data buffer will be associated with the reference file for further similarity comparison, thereby further improving the accuracy of the video image test;

[0082] 5. The image AI test module of the present invention can operate in a full test mode and a jump test mode, thereby being able to adapt to different events. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] The present invention will be further described below in conjunction with the accompanying drawings:

[0084] Figure 1 It is an internal processing block diagram of the video image testing element of the present invention;

[0085] Figure 2 A schematic diagram of the process of thread and queue running events, query and data buffer in the Gstreamer multimedia framework of the present invention;

[0086] Figure 3 It is a flow chart of the interaction between threads and queues of the present invention;

[0087] Figure 4 It is a flow chart of the whole test mode and the skip test mode of the present invention;

[0088] Figure 5 It is a similarity statistical diagram of the test result that the video image has no abnormality in the full test mode of the present invention;

[0089] Figure 6 It is a similarity statistical diagram of the test result of the video image being abnormal under the full test mode of the present invention;

[0090] Figure 7 It is a similarity statistical diagram of a video image without abnormality in a test result under the skip test mode of the present invention;

[0091] Figure 8 It is a similarity statistical diagram of video image abnormality in the test result under the skip test mode of the present invention;

[0092] Fig. 9 It is a schematic diagram of a pipeline block diagram of the system of the present invention; DETAILED DESCRIPTION

[0093] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0094] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only embodiments of a part of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0095] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions.

[0096] Glossary:

[0097] Gstreamer multimedia framework: is a multimedia framework for creating streaming media applications.

[0098] Element: An element is a basic component of a pipeline. It can interact with the outside world through pads and represent its own characteristics or functions to other elements in the pipeline.

[0099] Pad: Pad is the connection channel between an element and the outside world, describing the media type that the element can handle.

[0100] Pipeline: Pipeline, by connecting the pads of different elements in sequence to form a media processing pipeline, so that the data can be processed normally by each element in the process of flowing through the pipeline, and finally realize specific multimedia functions.

[0101] Event: Event, the elements of the pipeline can communicate with each other through events.

[0102] query: query, pipeline elements can communicate with each other through queries.

[0103] SERIALIZED: Serialization. Events and queries with serialization (SERIALIZED) type in the Gstreamer multimedia framework need to be processed in the order in which they enter the pipeline.

[0104] DOWNSTREAM: Downstream, in the Gstreamer multimedia framework, events and queries with the downstream (DOWNSTREAM) type are sent from the source pad (src pad) of the element and transmitted to the end of the pipeline.

[0105] buffer: data buffer, generally refers to a part of memory space.

[0106] Thread: Thread is a basic execution unit in a computer, used to implement concurrent execution of multiple tasks.

[0107] seek: skip play, this function is usually used on multimedia players.

[0108] queue: Queue, used to store data buffers pushed by upstream components, serialized type events and serialized type queries.

[0109] GstValidate: A logic monitoring tool and a unique testing function of the Gstreamer multimedia framework.

[0110] Example 1

[0111] A system for video image testing components based on a Gstreamer multimedia framework includes the Gstreamer multimedia framework, which is used to create a streaming media application;

[0112] The Gstreamer multimedia framework includes elements and pads. Pads are the connection channels between elements and the outside world. By connecting the pads of different elements in sequence, a media processing pipeline can be formed. The process of data flowing through the pipeline can be processed normally by each element, and finally specific multimedia functions can be realized.

[0113] element includes a video image test element, which exists in the Gstreamer multimedia framework in the form of a plug-in;

[0114] Since the video image test component exists in the form of a plug-in in the Gstreamer multimedia framework, users can arbitrarily select different plug-ins to build a complex multimedia test pipeline according to test needs, thereby improving the simplicity and ease of use of this system;

[0115] The components include receiving pads and source pads, and the components in the pipeline are connected by pads, such as Fig. 9 As shown;

[0116] After the source pad of the element that needs to test the output video data is connected to the video image test component and the pad format negotiation between the components inside the pipeline is successful, the upstream component will push the data buffer filled with video image data into the video image test component one by one, and the image AI test module of the video image test component will perform data calculations and judge whether the image is normal based on the obtained image histogram distribution value.

[0117] The video image test component will not make any modifications to the input video data, thereby ensuring that the input video format and the output video format are consistent, and can be better compatible with a variety of test scenarios, including conventional network video streaming test scenarios, file reading, decoding and playback test scenarios, file reading, parsing and playback test scenarios, and video device image capture and playback test scenarios, etc., with better compatibility.

[0118] The video image test element includes a queue and a thread. The queue and the thread exist in the video image test element. Figure 1 As shown;

[0119] The queue is used to store data buffers, serialized events, and serialized queries pushed by upstream components;

[0120] Data buffers, serialized events, and serialized queries enter the queue as the contents of the queue, and threads are used to dequeue the contents of the queue one by one, such as Figure 1 shown.

[0121] After the upstream component pushes the data buffer, event, and query into the video image test component, the buffer, serialized event, and serialized query will be stored in the queue. If the event and query are non-serialized types, the video image test component needs to process them immediately.

[0122] The threads dequeue the contents one by one from the queue. If the dequeued contents are buffers, the dequeued buffers will be submitted to the similarity comparison submodule of the image AI test module for processing. Figure 1 As shown;

[0123] If the content of the dequeue is a serialized event or a serialized query, it is determined whether it needs to be pushed to the downstream component based on the processing result.

[0124] The video image test element includes an image AI test module, and the image AI test module exists in the video image test element, such as Figure 1 As shown;

[0125] The image AI test module is used to perform data calculations and determine whether the video image in the data buffer is normal based on the image histogram distribution values ​​obtained from the calculations.

[0126] Furthermore, the image AI test module includes a similarity comparison submodule and a report processing submodule. When the similarity comparison submodule of the image AI test module receives the buffer pushed by the thread, it performs data calculation on the buffer, and the data calculation can obtain the histogram distribution value of the buffer;

[0127] The similarity comparison submodule compares the histogram distribution value of the current buffer with the histogram distribution value calculated for the previous buffer, and finally pushes the comparison result to the report processing submodule, such as Figure 1 As shown;

[0128] The report processing submodule can save the received comparison results to the output file directory, and this file will be used as a reference file for the next round of testing. After the comparison results of the next round are pushed to the report processing submodule, if the comparison results correspond to the reference file data, the test results are normal, otherwise the test fails.

[0129] In the Gstreamer multimedia framework, threads are called and run in a loop through the thread pool inside the multimedia framework, and no resident loop function is required inside the thread;

[0130] The thread execution process and the upstream component push process are asynchronous, so the mutex lock must be acquired before accessing the critical resource queue.

[0131] After successfully acquiring the mutex lock, the status of the queue can be queried. Only when the queue is in a normal state can the thread enter or exit the queue.

[0132] The Gstreamer multimedia framework stipulates that some specific events and queries cannot be serialized for processing. Therefore, non-serialized events and queries cannot be pushed into the queue and need to be processed immediately by the video image test component and pushed to the downstream components in a timely manner.

[0133] The image AI test module can work in full test mode and jump test mode, and send test report files to testers;

[0134] The test report saves the comparison test results of the buffer histogram distribution values ​​of the input report processing submodule of the last completed test pipeline. When the test fails, an error message can be submitted to the pipeline vertically;

[0135] This embodiment is based on the example that the image AI test module is set in the full test mode:

[0136] The similarity comparison submodule calculates the histogram distribution value corresponding to each received data buffer and compares it with the histogram distribution value of the previous adjacent data buffer. If the report processing submodule detects that the histogram distribution value comparison result is within a certain threshold range, it can be considered that the data images of the two adjacent data buffers have a high similarity and there is no abnormality in the image, such as Figure 5 As shown;

[0137] If the report processing submodule detects that the histogram distribution value comparison result exceeds the threshold range, it is considered that the currently submitted data buffer image has an abnormality, such as Figure 6 As shown, the comparison result of the histogram distribution values ​​of buffer8 and the previous data buffer buffer7 exceeds the set threshold, and the test fails.

[0138] The implementation method of the video image test component based on the Gstreamer multimedia framework includes the following steps:

[0139] Step 1. In the plug-in loading stage of the Gstreamer multimedia framework, a video image test component is registered in the Gstreamer multimedia framework, so that the source pad of the component that needs to test the output video data is connected to the video image test component. The video image test component will create a queue and a thread when registering;

[0140] Step 2. The upstream component pushes the events, queries, and data buffers filled with video image data one by one into the video image test component;

[0141] Step 3. The data buffer of the video image test component must first enter the queue;

[0142] Enter the event and query of the video image test component. The video image test component needs to determine whether it needs to enter the queue according to the type of event and query;

[0143] Step 4. The thread dequeues the data buffers, events, and queries stored in the queue one by one;

[0144] The thread submits the dequeued data buffer to the image AI test module in the video image test element for data calculation, and obtains the test result based on the image histogram distribution value obtained by the data calculation. The obtained test result will be saved to the output file directory, and then the thread exits;

[0145] After the thread pushes the dequeued events and queries to downstream components, the thread exits.

[0146] Through the above steps, the upstream component pushes the data buffers filled with video image data into the video image test component one by one, and the image AI test module of the video image test component performs data calculations. The image AI test module obtains the image histogram distribution values ​​of each data buffer according to the calculation and then compares them, thereby realizing automated testing and verification of video image data, improving the accuracy, reproducibility and test efficiency of video image testing.

[0147] Before the data buffer entering the video image test component in Step 3 enters the queue, it needs to acquire the mutex lock first. After the data buffer successfully acquires the mutex lock, the queue status is queried, including the following situations:

[0148] A. If the query result shows that the queue is not full, the data buffer can enter the queue, and then send a wake-up signal to the thread, waiting to be dequeued by the thread;

[0149] B. If the query result is that the queue is full, the data buffer needs to release the mutex lock and go into sleep mode. After the data buffer is awakened, it needs to reacquire the mutex lock. Figure 2 , Figure 3 shown.

[0150] In Step 3, according to the type of event and query, such as Figure 2 The following scenarios may be included:

[0151] A. If the event entering the video image test component is a serialized event, the video image test component determines that the event needs to enter the queue;

[0152] B. If the event entering the video image test component is a non-serialized event, the video image test component needs to process the event immediately;

[0153] C. If the query entering the video image test component is a serialized query, the video image test component determines that the query needs to enter the queue;

[0154] D. If the query entering the video image test component is a non-serialized query, the video image test component needs to process the query immediately.

[0155] In Step 4, after the image AI test module starts to receive the data buffer pushed by the thread, it needs to first determine the current test mode, such as Figure 4 As shown;

[0156] In this embodiment, the image AI test module determines that the current test mode is in the full test mode. In the full test mode, the image AI test module performs the following test steps on the data buffer:

[0157] Step 4A.2. Determine whether the current data buffer is the first frame data buffer;

[0158] Step 4A.2.1. If the current data buffer is the first frame data buffer, directly determine that the current data buffer is normal and exit from the thread;

[0159] Step 4A.2.2. If the current data buffer is not the first frame data buffer, compare the histogram distribution value of the current data buffer with the histogram distribution value of the previous data buffer;

[0160] Step 4A.3. After the similarity comparison submodule obtains the comparison result of the histogram distribution value of this round according to Step 4A.2.2, it pushes the comparison result to the report processing submodule;

[0161] Step 4A.4. The report processing submodule saves the comparison results of the received histogram distribution values ​​in this round to the output file directory as a reference file for the next round of testing, and determines whether the current data buffer has passed the test based on whether the comparison results of the histogram distribution values ​​in this round exceed the threshold, such as Figure 4 shown.

[0162] In Step 4A.4, if the comparison result of the histogram distribution value in this round exceeds the threshold, the report processing submodule will submit an error message to the pipeline bus, determine that the current data buffer fails the test and exit it from the thread, such as Figure 4 As shown;

[0163] In Step 4A.4, if the comparison result of the histogram distribution value in this round is that it does not exceed the threshold, the report processing submodule will query the output file directory whether there is a reference file, including the following situations:

[0164] A. If the report processing submodule does not find the reference file in the output file directory, the report processing submodule determines that the current data buffer has passed the test and causes it to exit from the thread;

[0165] B. If the report processing submodule finds that there is a reference file in the output file directory, the report processing submodule will read the reference file and retrieve the comparison data based on the timestamp of the data buffer in the reference file. The retrieved comparison data is compared with the comparison result of the current round of histogram distribution values ​​to determine whether the current data buffer has passed the test. Figure 4 As shown;

[0166] If the comparison result obtained by comparing the retrieved comparison data with the comparison result of the current round of histogram distribution value is above the threshold, the report processing submodule will submit an error message to the pipeline bus, determine that the current data buffer fails the test and exit it from the thread;

[0167] If the comparison result obtained by comparing the retrieved comparison data with the comparison result of the current round of histogram distribution value is not beyond the threshold, the report processing submodule determines that the current data buffer has passed the test and exits from the thread, such as Figure 4 shown.

[0168] By comparing the comparison results of the current data buffer with the reference file of the previous round, the test results will not only be the similarity comparison results of the current video image test component running calculations, but also be data associated with the comparison results of the previous round in the reference file, further improving the accuracy of the test.

[0169] Example 2

[0170] This embodiment is based on the first embodiment and takes the image AI test module being set in the skip test mode as an example;

[0171] In the skip test mode, when the video image test component receives a skip event, at least one buffer will always be cached in the queue. This buffer will be compared with the histogram distribution value of the new data buffer pushed into the video image test component after the skip event ends. Only when the comparison result conforms to the law can the test be passed.

[0172] In the skip test mode, the image AI test module starts working only when a skip event occurs, and continuously monitors several data buffers;

[0173] The number of monitoring data buffers can be configured through the property setting interface.

[0174] When the jump event ends, the Gstreamer multimedia framework will relocate the jump position to obtain the source content data, and continue to push new buffer content to the video image test component; the video image test component will receive the new buffer and go through the queue entry, queue exit, calculation of histogram distribution values, comparison of histogram distribution values, and determination of test results similar to the full test mode, and save the comparison results to the output file directory.

[0175] The histogram distribution value comparison result described in this embodiment complies with the law, which means that the histogram distribution value comparison result reported by the report processing submodule satisfies the following two laws:

[0176] (1) After the jump operation occurs, the histogram distribution value comparison result of the first data buffer obtained by parsing the corresponding jump position of the media file and the last data buffer before the jump operation occurs is a value far less than 1, which indicates that the current comparison frame data has no similar correlation;

[0177] (2) After the skip operation occurs, the histogram distribution value comparison result of the second data buffer obtained by parsing the corresponding skip position of the media file and the first data buffer is a value very close to 1, which indicates that the current comparison frame data has similar correlation; the image AI test module will continue to monitor the buffers generated after the skip operation and ensure that the subsequent adjacent frame data have similar correlation;

[0178] That is Figure 7 As shown, the histogram distribution value comparison result of the first data buffer 10 obtained by parsing the corresponding jump position of the media file and the last data buffer 9 before the jump operation occurs is 0.321, which is a value far less than 1; the histogram distribution value comparison result of the second data buffer 11 obtained by parsing the corresponding jump position of the media file and the first data buffer 10 is 0.956, which is a value very close to 1; and the histogram distribution value comparison results of several data buffers generated successively thereafter are also values ​​very close to 1, so it can be determined that the jump operation is normal and there is no abnormality in the video image after the jump; on the contrary, if the above rules are not met, it can be determined that the jump operation is abnormal, such as Figure 8 shown.

[0179] In the skip test mode, when the report processing submodule detects that there is a valid reference file in the output file directory, it will automatically read the data, and then filter out the histogram distribution value comparison data of the data buffer corresponding to the previous round of test pipeline according to the timestamp of the current push data buffer, and perform value comparison, so as to enhance the reliability of the video image data test results;

[0180] In the jump test mode, the comparison results saved by the report processing submodule will only be used as temporary log files for the test user to read only. Only the saved files generated to the output file directory in the full test mode can be used as input reference files.

[0181] In the jump test mode, the image AI test module tests the data buffer in the following steps:

[0182] Step 4B.1. Cache at least one data buffer in the queue;

[0183] Step 4B.2. If a skipping event is detected, the image AI test module continues to monitor several data buffers;

[0184] Step 4B.3. Refresh the queue in the callback function of the jump event, drain the content in the queue, and submit the last data buffer pushed into the queue to the similarity comparison submodule;

[0185] Step 4B.4. The similarity comparison submodule calculates the histogram distribution value of the data buffer and caches the histogram distribution value result of the data buffer;

[0186] Step 4B.5. After the jump event ends, the Gstreamer multimedia framework relocates to the jump position to obtain the source content data, and continues to push new data buffers to the video image test component;

[0187] Step 4B.6. The similarity comparison submodule receives new data buffers one by one, calculates the histogram distribution value thereof and compares it with the histogram distribution value of the previous data buffer to obtain the comparison result;

[0188] Step 4B.7. Push the comparison results one by one to the report processing submodule, which saves the comparison results to the output file directory; the report processing submodule evaluates whether the comparison results of the histogram distribution values ​​conform to the rules and determines whether the video image after the jump is abnormal, such as Figure 4 shown.

[0189] In the skip test mode, if no skip event is detected, it is necessary to determine the number of data buffers that the image AI test module continuously monitors;

[0190] If the number of data buffers continuously monitored by the image AI test module is zero, the data buffer test in the queue is judged to have passed;

[0191] If the number of data buffers continuously monitored by the image AI test module is not zero, the test steps are as follows:

[0192] Step4C.1. The data buffers enter the similarity comparison submodule one by one;

[0193] Step 4C.2. The similarity comparison submodule calculates the histogram distribution value of the data buffer and caches the histogram distribution value result of the current data buffer;

[0194] Step 4C.3. Compare the histogram distribution value of the current data buffer with the histogram distribution value of the previous data buffer, and then send the comparison result to the report processing submodule;

[0195] Step 4C.4. The report processing submodule saves the comparison result of the received histogram distribution value to the output file directory and decrements the value of the monitored data buffer;

[0196] Step 4C.5. The report processing submodule determines whether an abnormality occurs in the current data buffer based on whether the comparison result of the histogram distribution value conforms to the law, such as Figure 4 shown.

[0197] In Step 4C.5, if the report processing submodule determines that the comparison result of the histogram distribution value does not conform to the regularity, an error message will be submitted to the pipeline bus, and it will be determined that the current data buffer is abnormal.

[0198] In Step 4C.5, if the report processing submodule determines that the comparison result of the histogram distribution value conforms to the rule, the report processing submodule will query the output file directory to see whether there is a reference file, including the following situations:

[0199] A. If the report processing submodule does not find the reference file in the output file directory, the report processing submodule determines that the current data buffer is tested normally;

[0200] B. If the report processing submodule finds that there is a reference file in the output file directory, the report processing submodule will read the reference file and retrieve the comparison data based on the timestamp of the data buffer in the reference file. It will compare the retrieved comparison data with the comparison result of the current round of histogram distribution values ​​to determine whether the current data buffer has passed the test.

[0201] If the comparison result obtained by comparing the retrieved comparison data with the comparison result of the histogram distribution value is inconsistent with the law, the report processing submodule will submit an error message to the pipeline bus and determine that the current data buffer is a test exception.

[0202] If the comparison result obtained by comparing the retrieved comparison data with the comparison result of the histogram distribution value is consistent with the law, the report processing submodule determines that the current data buffer is tested normally.

[0203] The above are only specific embodiments of the present invention, but the technical features of the present invention are not limited thereto. Any simple changes, equivalent replacements or modifications made based on the present invention to achieve basically the same technical effects are all included in the protection scope of the present invention.

Claims

1. Implementation method of video image test component based on Gstreamer multimedia framework, Includes steps: Step 1. In the plug-in loading stage of the Gstreamer multimedia framework, a video image test component is registered in the Gstreamer multimedia framework, so that the source pad of the component that needs to test the output video data is connected to the video image test component. The video image test component will create a queue and a thread when registering; Step 2. The upstream component pushes the events, queries, and data buffers filled with video image data one by one into the video image test component; Step 3. The data buffer of the video image test component must first enter the queue; Enter the event and query of the video image test component. The video image test component needs to determine whether it needs to enter the queue according to the type of event and query; The following situations are included: A. If the event entering the video image test component is a serialized event, the video image test component determines that the event needs to enter the queue; B. If the event entering the video image test component is a non-serialized event, the video image test component needs to process the event immediately; C. If the query entering the video image test component is a serialized query, the video image test component determines that the query needs to enter the queue; D. If the query entering the video image test component is a non-serialized query, the video image test component needs to process the query immediately; Step 4. The thread dequeues the data buffer, events, and queries stored in the queue one by one; The thread submits the dequeued data buffer to the image AI test module in the video image test component for data calculation, and obtains the test result based on the image histogram distribution value obtained by the data calculation. The obtained test result will be saved to the output file directory, and then the thread exits; after the thread pushes the dequeued events and queries to the downstream components, the thread exits; After the image AI test module starts receiving the data buffer pushed by the thread, it must first determine the current test mode; If the image AI test module determines that the current test mode is in the full test mode, the test steps for the data buffer are as follows: Step4A.

1. After the similarity comparison submodule receives the data buffer pushed by the thread, it calculates the histogram distribution value of the current data buffer and caches the histogram distribution value result of the current data buffer; Step4A.

2. Determine whether the current data buffer is the first frame data buffer; Step4A.2.

1. If the current data buffer is the first frame data buffer, directly determine that the current data buffer is normal and exit from the thread; Step4A.2.

2. If the current data buffer is not the first frame data buffer, compare the histogram distribution value of the current data buffer with the histogram distribution value of the previous data buffer; Step4A.

3. After the similarity comparison submodule obtains the comparison result of the histogram distribution value of this round according to Step4A.2.2, it pushes the comparison result to the report processing submodule; Step 4A.

4. The report processing submodule saves the comparison results of the received histogram distribution values ​​in this round to the output file directory as a reference file for the next round of testing, and determines whether the current data buffer has passed the test based on whether the comparison results of the histogram distribution values ​​in this round exceed the threshold.

2. The method for implementing the video image test element based on the Gstreamer multimedia framework according to claim 1, characterized in that: In Step 3, before the data buffer of the video image test component enters the queue, it needs to acquire the mutex lock first. After the data buffer successfully acquires the mutex lock, the status of the queue is queried, including the following situations: A. If the query result shows that the queue is not full, the data buffer can enter the queue, and then send a wake-up signal to the thread, waiting to be dequeued by the thread; B. If the query result shows that the queue is full, the data buffer needs to release the mutex lock and go into sleep, and the data buffer needs to reacquire the mutex lock after being awakened.

3. The method for implementing the video image test element based on the Gstreamer multimedia framework according to claim 1, characterized in that: In Step 4A.4, if the comparison result of the histogram distribution value in this round exceeds the threshold, the report processing submodule will submit an error message to the pipeline bus, determine that the current data buffer fails the test, and cause it to exit from the thread.

4. The method for implementing the video image test element based on the Gstreamer multimedia framework according to claim 1, characterized in that: In Step 4A.4, if the comparison result of the histogram distribution value in this round is that it does not exceed the threshold, the report processing submodule will query the output file directory to see if there is a reference file, including the following situations: A. If the report processing submodule does not find the reference file in the output file directory, the report processing submodule determines that the current data buffer has passed the test and causes it to exit from the thread; B. If the report processing submodule finds that there is a reference file in the output file directory, the report processing submodule will read the reference file and retrieve the comparison data based on the timestamp of the data buffer in the reference file. It will compare the retrieved comparison data with the comparison result of the current round of histogram distribution values ​​to determine whether the current data buffer has passed the test.

5. The method for implementing the video image test element based on the Gstreamer multimedia framework according to claim 4, characterized in that: If the comparison result obtained by comparing the retrieved comparison data with the comparison result of the current round of histogram distribution values ​​exceeds the threshold, the report processing submodule will submit an error message to the pipeline bus, determine that the current data buffer fails the test and cause it to exit from the thread.

6. The method for implementing the video image test element based on the Gstreamer multimedia framework according to claim 4, characterized in that: If the comparison result obtained by comparing the retrieved comparison data with the comparison result of the current round of histogram distribution values ​​is not above the threshold, the report processing submodule determines that the current data buffer has passed the test and exits the thread.

7. The method for implementing a video image test element based on a Gstreamer multimedia framework according to claim 1, characterized in that: If the image AI test module determines that the current test mode is in the jump test mode, the test steps for the data buffer are as follows: Step4B.

1. Cache at least one data buffer in the queue; Step4B.

2. If a jump event is detected, the image AI test module continues to monitor several data buffers; Step4B.

3. Refresh the queue in the callback function of the jump event, drain the contents of the queue, and submit the last data buffer pushed into the queue to the similarity comparison submodule; Step4B.

4. The similarity comparison submodule calculates the histogram distribution value of the data buffer and caches the histogram distribution value result of the data buffer; Step4B.

5. After the jump event ends, the Gstreamer multimedia framework repositions to the jump position to obtain the source content data, and continues to push new data buffers to the video image test element; Step4B.

6. The similarity comparison submodule receives new data buffers one by one, calculates its histogram distribution value and compares it with the histogram distribution value of the previous data buffer to obtain a comparison result; Step 4B.

7. Push the comparison results one by one to the report processing submodule, and the report processing submodule saves the comparison results to the output file directory; The report processing submodule determines whether the video image after the jump is abnormal based on whether the comparison result of the evaluation histogram distribution value conforms to the law.

8. The method for implementing the video image test element based on the Gstreamer multimedia framework according to claim 7, characterized in that: In the skip test mode, if no skip event is detected, it is necessary to determine the number of data buffers that the image AI test module continuously monitors.

9. The method for implementing the video image test element based on the Gstreamer multimedia framework according to claim 8, characterized in that: If the number of data buffers continuously monitored by the image AI test module is zero, the data buffer test in the queue is judged to have passed.

10. The method for implementing a video image test element based on a Gstreamer multimedia framework according to claim 8, characterized in that: If the number of data buffers continuously monitored by the image AI test module is not zero, the test steps are as follows: Step4C.

1. The data buffers enter the similarity comparison submodule one by one; Step4C.

2. The similarity comparison submodule calculates the histogram distribution value of the data buffer and caches the histogram distribution value result of the current data buffer; Step4C.

3. The histogram distribution value of the current data buffer is compared with the histogram distribution value of the previous data buffer, and then the comparison result is sent to the report processing submodule; Step4C.

4. The report processing submodule saves the comparison result of the received histogram distribution value to the output file directory, and decrements the value of the monitored data buffer; Step4C.

5. The report processing submodule determines whether there is an abnormality in the current data buffer based on whether the comparison result of the histogram distribution value conforms to the law.

11. The method for implementing a video image test element based on a Gstreamer multimedia framework according to claim 10, characterized in that: In Step 4C.5, if the report processing submodule determines that the comparison result of the histogram distribution value does not conform to the regularity, an error message will be submitted to the pipeline bus, and it will be determined that the current data buffer is abnormal.

12. The method for implementing a video image test element based on a Gstreamer multimedia framework according to claim 10, characterized in that: In Step 4C.5, if the report processing submodule determines that the comparison result of the histogram distribution value conforms to the rule, the report processing submodule will query the output file directory whether there is a reference file, including the following situations: A. If the report processing submodule does not find the reference file in the output file directory, the report processing submodule determines that the current data buffer is tested normally; B. If the report processing submodule finds that there is a reference file in the output file directory, the report processing submodule will read the reference file and retrieve the comparison data based on the timestamp of the data buffer in the reference file. It will compare the retrieved comparison data with the comparison result of the current round of histogram distribution values ​​to determine whether the current data buffer has passed the test.

13. The method for implementing a video image test element based on a Gstreamer multimedia framework according to claim 12, characterized in that: If the comparison result obtained by comparing the retrieved comparison data with the comparison result of the histogram distribution value is inconsistent with the law, the report processing submodule will submit an error message to the pipeline bus and determine that the current data buffer is a test exception.

14. The method for implementing a video image test element based on a Gstreamer multimedia framework according to claim 12, characterized in that: If the comparison result obtained by comparing the retrieved comparison data with the comparison result of the histogram distribution value is consistent with the law, the report processing submodule determines that the current data buffer is tested normally.

15. A system for video image testing components based on the Gstreamer multimedia framework, characterized in that: Includes the Gstreamer multimedia framework for creating streaming media applications; Elements, which are basic building blocks of pipelines and are used to interact with the outside world through pads to represent their own characteristics or functions to other elements in the pipeline; Pads are used to connect components with the outside world and describe the media types that components can process. Pipelines are streaming media processing pipelines formed by sequentially connecting pads of different components, and are used to enable data to be processed by various components as it flows through the pipeline. The component includes a video image test component, which exists in the Gstreamer multimedia framework in the form of a plug-in. The video image test component can be connected to the source pad of the component that needs to test the output video data, and then receive the data buffer filled with video image data pushed by the upstream component, test the data buffer to determine whether the video image in the data buffer is normal; the video image test component includes an image AI test module, which exists in the video image test component, and the image AI test module is used to perform data calculations, and determine whether the video image in the data buffer is normal according to the image histogram distribution value obtained by the calculation; the image AI test module includes a similarity comparison submodule and a report processing submodule, the similarity comparison submodule is used to calculate the received data buffer to obtain its histogram distribution value, and the similarity comparison submodule can also compare the histogram distribution value of the current data buffer with the histogram distribution value of the previous data buffer, and push the comparison result to the report processing submodule; the report processing submodule can save the comparison result as a reference file, and compare the current comparison result with the reference file of the previous round to obtain the test result.

16. The system of video image testing element based on Gstreamer multimedia framework according to claim 15, characterized in that: The video image test element includes a queue and a thread, which exist in the video image test element. The queue is used to store the data buffer, serialized events and serialized queries pushed by the upstream element; the thread is used to dequeue the contents in the queue one by one.

Images