Automatic test method, system and equipment for video conference system

Abstract

The embodiment of the invention discloses an automatic testing method, system and equipment for a video conference system. According to the technical scheme provided by the embodiment of the invention, adaptive network simulation control is carried out through the corresponding test case, and the audio and video data of the video conference system influenced by the network simulation control is acquired, so that the case test is carried out, and the influence of the network on the audio and video related parameters is determined; on one hand, end-to-end audios and videos of a recorded video conference system are scored so as to determine the audio and video quality. The test report is displayed through the comprehensive scoring result and the use case test result, the video conference test result can be visually reflected, the test process of the video conference system is simplified through automatic test, the test efficiency is improved, and the time cost and the labor cost are saved. In addition, according to the scheme, automatic testing of the conference system is carried out through the audio and video scoring result and the case testing result, and the testing condition of the video conference system can be comprehensively and accurately reflected.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular, to an automated testing method, system, and device for a video conferencing system. Background Art

[0002] Video conferencing refers to a meeting in which users located at two or more locations communicate face-to-face through communication devices and networks. During a video conference, multiple terminals can access the MCU through video conferencing software to achieve the mutual transmission of meeting content. When developing video conferencing software, software testing needs to be performed on the video conferencing software, such as testing aspects such as video encoding, audio encoding, audio-video negotiation protocols, system platforms, and network environments to ensure product quality.

[0003] However, due to the numerous application scenarios of video conferencing software, the manual testing method is time-consuming, the testing process is relatively cumbersome, and the input of time cost and labor cost is relatively high. Summary of the Invention

[0004] The embodiments of the present application provide an automated testing method, system, and device for a video conferencing system, which can perform automated testing on video conferencing software, improve the testing efficiency of video conferencing software, and solve the technical problem of the cumbersome and time-consuming testing process of video conferencing software.

[0005] In a first aspect, the embodiments of the present application provide an automated testing method for a video conferencing system, including:

[0006] Performing network simulation control on a device under test based on network simulation parameters of a test case. During the process of controlling the device under test to execute the meeting operations corresponding to the test case, real-time acquiring first audio-video data of the device under test, and performing result assertion based on the first audio-video data to obtain a test result of the use case;

[0007] Acquiring second audio-video data from the device under test based on a physical connection link, and scoring the second audio-video data based on a set scoring rule to obtain a scoring result;

[0008] Generating and displaying a test report based on the second audio-video data, the scoring result, and the test result of the use case.

[0009] It can be seen that, on the one hand, the present application performs adaptive network simulation control through corresponding test cases, obtains the audio and video data of the video conferencing system affected by the network simulation control for use case testing, and determines the influence of the network on the audio and video related parameters; on the other hand, it scores by recording the end-to-end audio and video of the video conferencing system to determine the audio and video quality. Furthermore, by presenting the test report through comprehensive scoring results and use case test results, the test results of the video conferencing can be intuitively reflected, the test process of the video conferencing system can be simplified through automated testing, the test efficiency can be improved, and the time cost and labor cost can be saved.

[0010] In addition, this solution performs automated testing of the conferencing system through the audio and video scoring results and use case test results respectively to determine the audio and video quality and use case execution status of the video conferencing system. By performing use case execution tests for different scenarios, the test conditions of the video conferencing system can be comprehensively and accurately reflected.

[0011] Furthermore, controlling the device under test to execute the conferencing operations corresponding to the test cases includes:

[0012] Constructing a controller based on the scenario information corresponding to the test case, and transmitting instructions and files to the device under test through the controller and the intranet-extranet penetration protocol to control the device under test to execute the conferencing operations corresponding to the test cases.

[0013] The connection between the controller and the device under test can be achieved through the intranet-extranet penetration protocol, improving the robustness of the automated testing.

[0014] Furthermore, performing result assertion based on the first audio and video data to obtain the use case test results, including:

[0015] Obtaining media parameters based on the first audio and video data, comparing the media parameters with the preset assertion criteria, and generating the use case test results.

[0016] Furthermore, obtaining the second audio and video data from the device under test based on the physical connection link, including:

[0017] Connecting the device under test based on the physical connection link, receiving the original audio and video data transmitted by the device under test, playing the original audio and video data and performing real-time recording to generate the second audio and video data.

[0018] Adopting the method of obtaining the audio and video data of the device under test through the physical connection link for recording and playing does not consume the performance and bandwidth of the device under test, and can minimize the impact on the performance of the device under test.

[0019] Furthermore, generating the second audio and video data further includes:

[0020] Add frame numbers to each video frame in the second audio-visual data, and add picture boundary identifiers to the video frames for video score calculation.

[0021] Furthermore, score the second audio-visual data based on the set scoring rules to obtain a scoring result, including:

[0022] Obtain a cut video from the video frames of the second audio-visual data based on the picture boundary identifier, calculate video quality parameters based on the cut video, frame numbers, and corresponding recording timestamps, and determine the video score of the second audio-visual data based on the video quality parameters.

[0023] By calculating video quality parameters frame by frame, the minimization of video scoring granularity can be achieved, reducing scoring errors.

[0024] Furthermore, score the second audio-visual data based on the set scoring rules to obtain a scoring result, including:

[0025] Obtain an audio file from the second audio-visual data and calculate the audio score of the audio file based on an objective speech quality assessment algorithm.

[0026] In a second aspect, an automated testing system for a video conferencing system provided by an embodiment of the present application includes a master control unit, a recording unit, an audio-visual source unit, a network simulation unit, a scoring unit, and a test report unit;

[0027] The network simulation unit is used to perform network simulation control on the device under test based on the network simulation parameters of the test case;

[0028] The master control unit is used to, during the process of controlling the device under test to execute the conference operations corresponding to the test case, obtain the first audio-visual data of the device under test in real time, perform result assertion based on the first audio-visual data, and obtain the test result of the use case;

[0029] The audio-visual source unit is used to connect to the device under test based on a physical connection link, receive the original audio-visual data transmitted by the device under test, and play the original audio-visual data;

[0030] The recording unit is used to perform real-time recording during the process of playing the original audio-visual data to generate second audio-visual data;

[0031] The scoring unit is used to score the second audio-visual data based on the set scoring rules to obtain a scoring result;

[0032] The test report unit is used to generate a test report based on the second audio-visual data, the scoring result, and the test result of the use case and display it.

[0033] In a third aspect, an automated testing device for a video conferencing system provided by an embodiment of the present application includes:

[0034] A use case testing module, configured to perform network simulation control on a device under test based on network simulation parameters of test cases. During the process of controlling the device under test to execute the conference operations corresponding to the test cases, it obtains the first audio-visual data of the device under test in real time, and performs result assertion based on the first audio-visual data to obtain the use case test result;

[0035] A scoring module, configured to obtain second audio-visual data from the device under test based on a physical connection link, and score the second audio-visual data based on a set scoring rule to obtain a scoring result;

[0036] A display module, configured to generate and display a test report based on the second audio-visual data, the scoring result, and the use case test result.

[0037] In a fourth aspect, an embodiment of the present application provides an electronic device, including:

[0038] A memory and one or more processors;

[0039] The memory is configured to store one or more programs;

[0040] When the one or more programs are executed by the one or more processors, the one or more processors implement the automated test method for a video conferencing system as described in the first aspect.

[0041] In a fifth aspect, an embodiment of the present application provides a storage medium containing computer-executable instructions, and the computer-executable instructions are used to execute the automated test method for a video conferencing system as described in the first aspect when executed by a computer processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a flowchart of an automated test method for a video conferencing system provided in Embodiment 1 of the present application;

[0043] Figure 2 is a schematic structural diagram of an automated test system for a video conferencing system provided in Embodiment 1 of the present application;

[0044] Figure 3 is a schematic connection diagram between a master control unit and a device under test in Embodiment 1 of the present application;

[0045] Figure 4 is a flowchart of an automated test for a video conferencing system in Embodiment 1 of the present application;

[0046] Figure 5 is a schematic structural diagram of an automated test device for a video conferencing system provided in Embodiment 2 of the present application;

[0047] Figure 6It is a schematic structural diagram of an electronic device provided in Embodiment 3 of the present application. Detailed implementation manners

[0048] In order to make the objectives, technical solutions and advantages of the present application clearer, the following further describes the specific embodiments of the present application in detail with reference to the accompanying drawings. It can be understood that the specific embodiments described herein are only used to explain the present application, rather than limiting the present application. Additionally, it should be noted that, for the sake of convenience of description, only parts related to the present application are shown in the drawings rather than all the content. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations (or steps) as sequential processes, many of the operations can be implemented in parallel, concurrently or simultaneously. In addition, the order of the operations can be rearranged. When the operations are completed, the process can be terminated, but there can also be additional steps not included in the drawings. The process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0049] An automated test method for a video conferencing system provided by the present application aims to achieve the automated test of the video conferencing system and improve the test efficiency of the video conferencing system.

[0050] In the test scenario of a video conferencing system, due to the large number of test scenarios and corresponding use cases, the manual test method is usually adopted to execute the use case tests one by one for each scenario. The manual test method is very time-consuming. It takes dozens of days to complete a variable combination test that affects the audio and video quality of the video conference. Moreover, there are many test index parameters that are difficult to directly obtain in manual testing, such as video frame interval, video frame structure similarity, audio and video objective scoring, etc. If the test report requires these difficult-to-obtain test index parameters, the test cycle needs to be further extended. In addition, the work repetition degree in the process of manually generating a test report is high, and the test personnel need to manually synchronize the test process to all parties. The process of closing the quality test loop is cumbersome and time-consuming.

[0051] Based on this, an automated testing method for the video conferencing system according to the embodiments of the present application is provided to solve the technical problem of the cumbersome and time-consuming testing process of video conferencing software. By performing adaptive network simulation control on the corresponding test cases, the audio and video data affected by the network simulation control of the video conferencing system is obtained for use case testing to determine the influence of the network on the audio and video related parameters. On the one hand, the end-to-end audio and video of the video conferencing system are recorded for scoring to determine the audio and video quality. Furthermore, by presenting the test report based on the comprehensive scoring results and use case testing results, the test results of the video conferencing can be intuitively reflected. The testing process of the video conferencing system is simplified through automated testing, improving the testing efficiency and saving time and labor costs. In addition, this solution performs automated testing of the conferencing system through the audio and video scoring results and use case testing results respectively to determine the audio and video quality and use case execution status of the video conferencing system. By performing use case execution testing for different scenarios, the testing situation of the video conferencing system can be comprehensively and accurately reflected.

[0052] Embodiment 1:

[0053] Figure 1 The flowchart of an automated testing method for a video conferencing system provided by Embodiment 1 of the present application is given. The automated testing method for the video conferencing system provided in this embodiment can be executed by the automated testing device of the video conferencing system. The automated testing device of the video conferencing system can be implemented in software and / or hardware. The automated testing device of the video conferencing system can be composed of two or more physical entities or a single physical entity. Generally speaking, the automated testing device of the video conferencing system can be a computing device such as a computer.

[0054] The following takes the automated testing device of the video conferencing system as the main body for executing the automated testing method of the video conferencing system as an example for description. Referring to Figure 1 , the automated testing method of the video conferencing system specifically includes:

[0055] S110. Perform network simulation control on the device under test based on the network simulation parameters of the test case. During the process of controlling the device under test to execute the conference operation corresponding to the test case, the first audio and video data of the device under test is obtained in real time, and result assertions are made based on the first audio and video data to obtain the use case test results;

[0056] S120. Obtain the second audio and video data from the device under test based on the physical connection link, and score the second audio and video data based on the set scoring rules to obtain the scoring results;

[0057] S130. Generate and display a test report based on the second audio and video data, the scoring results, and the use case test results.

[0058] When this application conducts tests on a video conferencing system, it comprehensively performs automated tests on the video conferencing system through use case testing and audio-video scoring. Among them, use case testing can determine the execution status of corresponding use cases, and audio-video scoring can determine the end-to-end audio-video quality. Based on this, corresponding use case automated testing and audio-video quality testing can be carried out for different scenarios, and a test report can be automatically generated, thereby improving the testing efficiency.

[0059] Among them, the automated testing of the video conferencing system is carried out through an automated testing system for a video conferencing system, referring to Figure 2 , the automated testing system for the video conferencing system includes a master control unit, a recording unit, an audio-video source unit, a network simulation unit, a scoring unit, and a test report unit; among them, the network simulation unit is used to perform network simulation control on the device under test based on the network simulation parameters of the test case; the master control unit is used to obtain the first audio-video data of the device under test in real time during the process of controlling the device under test to execute the corresponding meeting operation of the test case, and perform result assertion based on the first audio-video data to obtain the use case test result; the audio-video source unit is used to connect to the device under test based on a physical connection link, receive the original audio-video data transmitted by the device under test, and play the original audio-video data; the recording unit is used to perform real-time recording during the process of playing the original audio-video data to generate second audio-video data; the scoring unit is used to score the second audio-video data based on the set scoring rules to obtain a scoring result; the test report unit is used to generate and display a test report based on the second audio-video data, the scoring result, and the use case test result.

[0060] Specifically, the connection relationship (including physical connection and network connection) between each unit and the device under test is as Figure 2 shown. The master control unit is connected to the device under test (terminal / MCU) through the network, and the recording unit is connected to the test report unit; the recording unit is physically connected to the device under test through the network (physically, the terminal is connected to the recording unit through an HDMI cable to output audio-video data), and is connected to the audio-video source unit, the network simulation unit, the scoring unit, and the test report unit through the network; the audio-video source unit is physically connected to the device under test (the audio-video source unit is connected to the terminal under test through an HDMI cable to input audio-video data); the network simulation unit is physically and network-connected to the device under test (the network simulation unit is connected to the terminal under test through a physical network cable to control its network to achieve the purpose of simulating the network); the scoring unit is connected to the recording unit through the network.

[0061] Based on the above automated testing system, when performing video automated testing, the audio-video scoring result and the use case test result can be determined respectively, and then a test report can be generated and displayed.

[0062] Specifically, when starting the conference system test, network simulation control is first performed through the network simulation unit to test the execution of use cases under the influence of network simulation. The network simulation unit includes a dynamic network simulation module. The so-called dynamic network simulation means that different network parameter changes are achieved at different times. Specifically, the network parameters mainly include the following: network bandwidth, latency, packet loss rate, packet error rate, and out-of-order rate. According to the currently tested use case, the network simulation unit can control the network of the device under test in the simulated network environment to undergo specified network parameter changes. For example, it is required that all data packets have a 200-millisecond latency within 2 - 10 seconds in real time, and a packet loss with a 20% packet loss rate occurs within 11 - 20 seconds. Through the control of these network parameter changes, the real network transmission situation between the devices under test can be simulated, and thus accurate use case testing can be achieved.

[0063] After completing the network simulation test, the master control unit controls the device under test to execute the conference operations corresponding to the test use case. During this process, the audio and video data transmitted between the devices under test can be collected for use case testing. The master control unit is the control center of the entire automated test system. Its main function is to control the devices under test and the MCU required for each scenario, execute various conference operations required by the use case, and control the recording unit to record and score the audio and video of the device under test according to multiple scenarios and corresponding use cases input by the user using this automated test system, and report the use case execution results to the test report unit.

[0064] Among them, the master control unit controls the device under test to execute the conference operations corresponding to the test use case, including:

[0065] Construct a controller based on the scenario information corresponding to the test use case, and transmit instructions and files to the device under test through the controller and the NAT traversal protocol to control the device under test to execute the conference operations corresponding to the test use case.

[0066] The master control unit includes a control engine module and a controller module. It is mainly responsible for parsing the input scenario information and use cases, inputting the scenario information to the controller module to construct a controller for the corresponding device, and controlling each device to execute various test operations according to the controller. The controller module is mainly responsible for creating a controller that can control each device according to the input scenario information and the configured information of each device. The interfaces of each controller are unified and interact using the RPC protocol. Optionally, for different application platforms (Android platform, Windows platform), controllers for different platforms can be configured.

[0067] In addition, such as Figure 3As shown in the figure, for a device under a simulation network, an internal and external network penetration module is configured inside its controller, enabling the master control unit and the device in the simulation network to transmit instructions and files through the internal and external network penetration module. Since network simulation requires devices that need the simulation network to be in a specific local area network, at this time, the controlled device cannot directly establish a network connection with the master control unit in the external network. At this time, by using the internal and external network penetration module, the master control unit and the device in the simulation network can establish a network connection through the internal and external network penetration protocol to transmit instructions and files. In this way, the connection between the controller and the device to be tested is realized through the internal and external network penetration protocol, improving the robustness of automated testing.

[0068] After the master control unit controls the device to be tested to perform relevant conference operations, the audio data between the devices to be tested can be collected in real time, and this audio data is defined as the first audio data. It can be understood that the first audio data is affected by network simulation control. Therefore, the execution of use cases under the network simulation effect can be tested, and then result assertions can be made to generate corresponding use case test results.

[0069] Among them, based on the first audio-visual data, result assertions are made to obtain the use case test results, including:

[0070] Media parameters are obtained based on the first audio-visual data, and the media parameters are compared with the preset assertion standards to generate use case test results.

[0071] For example, assuming that the encoding frame rate configured in a use case is 30 frames, then the assertion standard is 30 frames, and the frame rate of the actually received first audio-visual data is 15 frames. At this time, the assertion fails and the corresponding use case test results are output. And so on, by setting different assertion standards, the first audio-visual data is collected during the execution of the use case. According to its media parameters, such as the statistical results of media parameters such as encoding format, resolution, frame rate, and bit rate, the use case test results can be determined by comparing the corresponding assertion standards.

[0072] On the other hand, for the determination of the audio-visual scoring results, it is determined by the end-to-end audio-visual data provided by the recording unit. It should be noted that different from the above first audio-visual data, the second audio-visual data is used to test the quality of the audio-visual itself. Therefore, the influence of the network does not need to be considered. Therefore, in the embodiments of the present application, the device to be tested is directly connected through a physical connection link to obtain the second audio-visual data.

[0073] Among them, obtaining the second audio-visual data from the device to be tested based on the physical connection link includes:

[0074] Connect the device to be tested based on the physical connection link, receive the original audio-visual data transmitted by the device to be tested, play the original audio-visual data and perform real-time recording to generate the second audio-visual data.

[0075] The recording unit is connected to the device under test through the physical connection link of the audio - video source unit, and then obtains the original audio - video data from the device under test. Then, under the control of the recording unit, the corresponding audio - video data is played. During this process, the recording unit records the original audio - video data in real - time to generate second audio - video data.

[0076] Further, generating the second audio - video data further includes:

[0077] Frame numbers are added to each video frame in the second audio - video data one by one, and picture boundary identifiers are added to the video frames for video score calculation.

[0078] By adding consecutive frame numbers to each frame of the video, it is used for the subsequent scoring unit to perform OCR recognition of each frame number. And a white stripe is added to the boundary of each video frame for the subsequent scoring unit to perform video frame positioning. In this way, the video and audio data of the video conference are directly output to the recording device of the recording unit through the HDMI cable, so that the audio - video of the device under test is directly output to the recording unit for recording, realizing the minimization of the audio - video scoring granularity (in units of frames), and realizing the minimization of the impact on the performance of the device under test (without consuming the device's network bandwidth and processor performance for additional audio - video transmission for recording).

[0079] After that, the scoring unit can score based on the second audio - video data. Scoring the second audio - video data based on the set scoring rules to obtain a scoring result, including:

[0080] An audio file is obtained from the second audio - video data, and the audio score of the audio file is calculated based on the objective speech quality assessment algorithm.

[0081] A cut video is obtained from the video frames of the second audio - video data based on the picture boundary identifier, video quality parameters are calculated based on the cut video, frame numbers, and the corresponding recording timestamps, and the video score of the second audio - video data is determined based on the video quality parameters.

[0082] Among them, for the audio score of the second audio - video data, first, the audio file of the second audio - video data recorded by the recording unit is obtained, then the audio file is read and the PESQ algorithm (objective speech quality assessment algorithm) is used for audio scoring. The score is between - 0.5 and 4.5. The higher the PESQ score, the higher the audio score, and then it is output to the scoring result data structure.

[0083] For the video scoring of the second audio - video data, first obtain the original video played by the audio - video source, the video recorded by the second audio - video data, and the recording timestamp corresponding to each frame. Then, use the white border strips in the video for positioning and use the OCR algorithm to identify the pre - made frame numbers in the video. Cut the video according to the identified positioning and combine it into a new cut video. Furthermore, use the cut video, the identified frame numbers, and the recording timestamps to calculate video test quality parameters such as the time information, spatial information, structural similarity, multi - method evaluation fusion score of video quality, frame rate, freezing time, and latency for each frame, and characterize the video score with any one or a combination of these video test quality parameters. Then, compress the recorded audio and video, and organize and format the scoring results of the audio and video, and report them to the test report unit. In this way, by calculating the video quality parameters frame by frame, the minimization of the video scoring granularity can be achieved, reducing the scoring error.

[0084] Finally, based on the above - mentioned second audio - video data, scoring results, and use - case test results, the test report unit can present the test results of the conferencing system to the user in a web form. The test report unit can classify each reported use - case test result and scoring result according to the pre - filled marking strings, judge whether the use - case passes, present the audio - video during the test process to the user on the web, and display the detailed results of each frame in the form of a line chart. In addition, the test report unit can also provide functions such as displaying use - cases to be processed by the user, adding remarks to each use - case, switching the use - case status, switching the use - case handler, and rerunning the automated test, realizing the closed - loop processing of the video conferencing system test.

[0085] Optionally, during the test process of the video conferencing system, if the executed test type is protocol negotiation test, the recording unit is not required to participate. At this time, the master control unit obtains the audio - video media data of the device to be tested, and directly reports the use - case test results to the test report unit after making result assertions according to the use - cases. According to actual needs, the master control unit can expand the test types and access more different types of tests without changing the architecture. And there is no need for physical wiring, and it can be deployed on any PC device under the same network segment, making the trigger of a round of automated test more convenient.

[0086] By providing the automated test system for this video conferencing system, various test scenarios and test cases composed of variables affecting the audio - video quality of the video conferencing can be automatically executed for quality testing, report generation, etc. And provide a web platform to improve the efficiency of the quality - testing closed - loop. Each automated test report can contain multiple quality - testing items and test metric parameters that are difficult to obtain through manual testing, greatly shortening the time required for a round of testing.

[0087] Exemplarily, such as Figure 4As shown, when testers conduct tests on the conferencing system, they use the automated testing system of the video conferencing system according to the embodiments of the present application. By inputting the scenarios that the system currently needs to test and the corresponding use cases, a round of testing process is triggered. The master control unit controls the recording unit to start the testing process. After the testing process starts, the recording unit first controls the network simulation unit according to the network simulation parameters in the use case. The network simulation unit starts to control various network metrics of the upstream or downstream network of the corresponding device, such as packet loss, latency, bandwidth, etc., to achieve the purpose of network simulation. The master control unit controls the terminals and MCUs of each platform to start a meeting according to the input scenario and execute the operations that need to be performed during the meeting described in the test case, and then controls the recording unit to start executing subsequent operations. At this time, after obtaining the audio and video media data of the device to be tested, result assertions are made according to the use case, and then the test results of the use case are directly reported to the test report unit. On the other hand, the recording unit controls the audio and video source unit to play the corresponding mode of audio and video, and starts to record the audio and video required by the terminal device. After completion, it first notifies the master control unit, and then controls the scoring unit to start executing subsequent operations. The master control unit continuously obtains the media data related to the audio and video of the tested terminal / MCU during the recording process of the recording unit. After receiving the callback notification of the recording completion from the recording unit, it makes assertions on the obtained media data according to the use case and reports the execution results of the use case. The scoring unit starts the corresponding scoring after obtaining the audio and video recorded by the recording unit and asynchronously returns the results to the recording unit. After receiving the asynchronous scoring results from the scoring unit, the recording unit reports them to the test report unit. After receiving the execution results of the use case from the master control unit, the test report unit will first save them. After the asynchronous scoring results of the corresponding use case are reported, it combines the execution results of the use case and the scoring results to calculate the final test results of the use case, saves them, and displays all data on the web. Testers or R & D personnel access the test report unit through the web to obtain the test report for a corresponding round of testing, analyze the failed items, and through functions such as adding remarks, switching the use case status, and switching the use case handlers, the test report unit will notify the corresponding R & D personnel to handle the failed items, realizing a testing closed-loop.

[0088] As described above, through adaptive network simulation control corresponding to test cases, audio and video data affected by network simulation control of the video conferencing system are obtained for use case testing to determine the impact of the network on audio and video related parameters. On the one hand, the end-to-end audio and video of the video conferencing system are recorded for scoring to determine the audio and video quality. Furthermore, by presenting the test report based on the comprehensive scoring results and use case test results, the test results of the video conferencing can be intuitively reflected. The test process of the video conferencing system is simplified through automated testing, improving the test efficiency and saving time and labor costs. In addition, in this solution, the automated testing of the conferencing system is respectively carried out through the audio and video scoring results and use case test results to determine the audio and video quality and use case execution status of the video conferencing system. By performing use case execution tests corresponding to different scenarios, the test situation of the video conferencing system can be comprehensively and accurately reflected.

[0089] Embodiment 2:

[0090] Based on the above embodiment, Figure 5 FIG. is a schematic structural diagram of an automated testing apparatus for a video conferencing system provided in Embodiment 2 of the present application. Refer to Figure 5 The automated testing apparatus for a video conferencing system provided in this embodiment specifically includes: a use case testing module 31, a scoring module 32, and a presentation module 33.

[0091] Among them, the use case testing module 31 is used to perform network simulation control on the device under test based on the network simulation parameters of the test case. During the process of controlling the device under test to execute the conference operations corresponding to the test case, the first audio and video data of the device under test are obtained in real time, and result assertions are made based on the first audio and video data to obtain the use case test results;

[0092] The scoring module 32 is used to obtain the second audio and video data from the device under test based on the physical connection link, and score the second audio and video data based on the set scoring rules to obtain the scoring results;

[0093] The presentation module 33 is used to generate and present a test report based on the second audio and video data, the scoring results, and the use case test results.

[0094] As described above, through adaptive network simulation control corresponding to test cases, audio and video data affected by network simulation control of the video conferencing system are obtained for use case testing to determine the impact of the network on audio and video related parameters. On the one hand, by recording the end-to-end audio and video of the video conferencing system for scoring to determine the audio and video quality. Furthermore, by presenting the test report through the comprehensive scoring results and use case test results, the test results of the video conferencing can be intuitively reflected, simplifying the test process of the video conferencing system through automated testing, improving the test efficiency, and saving time and labor costs. In addition, this solution conducts automated testing of the conferencing system through the audio and video scoring results and use case test results respectively to determine the audio and video quality and use case execution status of the video conferencing system. By performing use case execution tests corresponding to different scenarios, the test conditions of the video conferencing system can be comprehensively and accurately reflected.

[0095] Specifically, controlling the device under test to execute the meeting operations corresponding to the test cases includes:

[0096] Constructing a controller based on the scenario information corresponding to the test case, and transmitting instructions and files to the device under test through the controller and the intranet-extranet penetration protocol to control the device under test to execute the meeting operations corresponding to the test case.

[0097] Through the intranet-extranet penetration protocol, the connection between the controller and the device under test can be realized, improving the robustness of automated testing.

[0098] Specifically, based on the first audio and video data, result assertions are made to obtain the use case test results, including:

[0099] Obtaining media parameters based on the first audio and video data, comparing the media parameters with the preset assertion criteria, and generating use case test results.

[0100] Specifically, based on the physical connection link, the second audio and video data is obtained from the device under test, including:

[0101] Connecting the device under test based on the physical connection link, receiving the original audio and video data transmitted by the device under test, playing the original audio and video data and performing real-time recording to generate the second audio and video data.

[0102] Adopting the method of obtaining, recording and playing the audio and video data of the device under test through the physical connection link does not consume the bandwidth of the device under test and can minimize the impact on the performance of the device under test.

[0103] Specifically, generating the second audio and video data further includes:

[0104] Adding frame numbers to each video frame in the second audio and video data and adding picture boundary identifiers to the video frames for video scoring calculation.

[0105] Specifically, score the second audio-video data based on the set scoring rules to obtain a scoring result, including:

[0106] Obtain a cut video from the video frames of the second audio-video data based on the picture boundary identifier, calculate the video quality parameter based on the cut video, the frame number, and the corresponding recording timestamp, and determine the video score of the second audio-video data based on the video quality parameter.

[0107] By calculating the video quality parameter frame by frame, the minimization of the video scoring granularity can be achieved, and the scoring error can be reduced.

[0108] Specifically, score the second audio-video data based on the set scoring rules to obtain a scoring result, including:

[0109] Obtain an audio file from the second audio-video data, and calculate the audio score of the audio file based on the objective speech quality assessment algorithm.

[0110] The automated testing device for the video conferencing system provided in the second embodiment of the present application can be used to execute the automated testing method for the video conferencing system provided in the first embodiment, and has the corresponding functions and beneficial effects.

[0111] Embodiment 3:

[0112] The third embodiment of the present application provides an electronic device. Referring to Figure 6 , the electronic device includes: a processor 31, a memory 32, a communication module 33, an input device 34, and an output device 35. The number of processors in the electronic device can be one or more, and the number of memories in the electronic device can be one or more. The processor, memory, communication module, input device, and output device of the electronic device can be connected through a bus or other means.

[0113] The memory, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as program instructions / modules corresponding to the automated testing method of the video conferencing system described in any embodiment of the present application (for example, the use case testing module, scoring module, and display module in the automated testing device of the video conferencing system). The memory mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store data created according to the use of the device, etc. In addition, the memory can include high-speed random access memory, and can also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage devices. In some instances, the memory can further include a memory remotely set relative to the processor, and these remote memories can be connected to the device through a network. Examples of the above network include but are not limited to the Internet, enterprise intranet, local area network, mobile communication network, and their combinations.

[0114] The communication module is used for data transmission.

[0115] The processor executes various functional applications and data processing of the device by running the software programs, instructions, and modules stored in the memory, that is, implements the above-mentioned automated testing method of the video conferencing system.

[0116] The input device can be used to receive input digital or character information, and generate key signal inputs related to the user settings and function control of the device. The output device can include display devices such as a display screen.

[0117] The above-provided electronic device can be used to execute the automated testing method of the video conferencing system provided in the first embodiment above, and has corresponding functions and beneficial effects.

[0118] Embodiment 4:

[0119] The embodiment of the present application also provides a storage medium containing computer-executable instructions, and the computer-executable instructions are used to execute an automated testing method of a video conferencing system when executed by a computer processor. The automated testing method of the video conferencing system includes: performing network simulation control on a device under test based on network simulation parameters of a test case, and during the process of controlling the device under test to execute the conference operations corresponding to the test case, real-time acquiring first audio-video data of the device under test, performing result assertion based on the first audio-video data to obtain a use case test result; acquiring second audio-video data from the device under test based on a physical connection link, and scoring the second audio-video data based on a set scoring rule to obtain a scoring result; generating a test report based on the second audio-video data, the scoring result, and the use case test result and performing display.

[0120] Storage medium - Any of various types of memory devices or storage devices. The term "storage medium" is intended to include: installation media such as CD-ROMs, floppy disks or magnetic tape devices; computer system memory or random access memory such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory such as flash memory, magnetic media (such as hard disks or optical storage); registers or other similar types of memory elements, etc. The storage medium may also include other types of memory or combinations thereof. Additionally, the storage medium may be located in a first computer system in which the program is executed, or may be located in a different second computer system that is connected to the first computer system via a network (such as the Internet). The second computer system may provide program instructions to the first computer for execution. The term "storage medium" may include two or more storage media residing in different locations (such as in different computer systems connected via a network). The storage medium may store program instructions executable by one or more processors (such as embodied as a computer program).

[0121] Of course, for a storage medium containing computer-executable instructions provided by an embodiment of the present application, the computer-executable instructions are not limited to the automated test method of the video conferencing system as described above, and can also perform related operations in the automated test method of the video conferencing system provided by any embodiment of the present application.

[0122] The automated test device, storage medium, and electronic device of the video conferencing system provided in the above embodiments can execute the automated test method of the video conferencing system provided by any embodiment of the present application. For technical details not described in detail in the above embodiments, reference can be made to the automated test method of the video conferencing system provided by any embodiment of the present application.

[0123] The above is only the preferred embodiment of the present application and the technical principles applied. The present application is not limited to the specific embodiments described herein. Various obvious changes, re-adjustments, and substitutions that can be made by those skilled in the art will not depart from the protection scope of the present application. Therefore, although the present application has been described in relatively detail through the above embodiments, the present application is not limited to the above embodiments. Without departing from the concept of the present application, it may include more other equivalent embodiments, and the scope of the present application is determined by the scope of the claims.

Claims

1. An automated testing method for a video conferencing system, characterized in that, Including: Based on the network simulation parameters of the test case, perform network simulation control on the device under test. During the process of controlling the device under test to execute the conference operation corresponding to the test case, obtain the first audio-visual data of the device under test in real time, and perform result assertion based on the first audio-visual data to obtain the test result of the use case; Obtain the second audio-visual data from the device under test based on the physical connection link, and score the second audio-visual data based on the set scoring rules to obtain the scoring result; Generate a test report based on the second audio-visual data, the scoring result, and the test result of the use case and display it.

2. The automated testing method for a video conferencing system according to claim 1, wherein The controlling the device under test to execute the conference operation corresponding to the test case includes: Construct a controller based on the scenario information corresponding to the test case, and transmit instructions and files to the device under test through the controller and the intranet-extranet penetration protocol to control the device under test to execute the conference operation corresponding to the test case.

3. The automated testing method for a video conferencing system according to any one of claims 1-2, characterized in that, The performing result assertion based on the first audio-visual data to obtain the test result of the use case includes: Obtain media parameters based on the first audio-visual data, compare the media parameters with the preset assertion criteria, and generate the test result of the use case.

4. The automated test method for a video conferencing system according to claim 1, characterized in that The obtaining the second audio-visual data from the device under test based on the physical connection link includes: Connect to the device under test based on the physical connection link, receive the original audio-visual data transmitted by the device under test, play the original audio-visual data and perform real-time recording to generate the second audio-visual data.

5. The automated test method for a video conferencing system according to claim 4, wherein The generating the second audio-visual data further includes: Add frame numbers to each video frame in the second audio-visual data one by one, and add picture boundary identifiers to the video frames for video scoring calculation.

6. The automated test method for a video conferencing system according to claim 5, wherein The scoring the second audio-visual data based on the set scoring rules to obtain the scoring result includes: Obtain the cut video from the video frames of the second audio-visual data based on the picture boundary identifier, calculate the video quality parameters based on the cut video, the frame number, and the corresponding recording timestamp, and determine the video score of the second audio-visual data based on the video quality parameters.

7. The automated test method for a video conferencing system according to claim 6, wherein The scoring the second audio-visual data based on the set scoring rules to obtain the scoring result includes: Obtain the audio file from the second audio-visual data, and calculate the audio score of the audio file based on the objective speech quality evaluation algorithm.

8. An automated test system for a video conferencing system, characterized in that, Including a total control unit, a recording unit, an audio-visual source unit, a network simulation unit, a scoring unit, and a test report unit; The network simulation unit is used to perform network simulation control on the device under test based on the network simulation parameters of the test case; The total control unit is used to obtain the first audio-visual data of the device under test in real time during the process of controlling the device under test to execute the conference operation corresponding to the test case, and perform result assertion based on the first audio-visual data to obtain the test result of the use case; The audio-visual source unit is used to connect to the device under test based on the physical connection link, receive the original audio-visual data transmitted by the device under test, and play the original audio-visual data; The recording unit is used to perform real-time recording during the process of playing the original audio-visual data to generate the second audio-visual data; The scoring unit is configured to score the second audio - video data based on a set scoring rule to obtain a scoring result; The test report unit is configured to generate and display a test report based on the second audio - video data, the scoring result, and the test case test result.

9. An automated test device for a video conferencing system, characterized in that, Comprising: A test case test module, configured to perform network simulation control of a device under test based on network simulation parameters of a test case. During the process of controlling the device under test to execute the conference operation corresponding to the test case, the first audio - video data of the device under test is obtained in real - time, and result assertion is performed based on the first audio - video data to obtain a test case test result; A scoring module, configured to obtain second audio - video data from a device under test based on a physical connection link, and score the second audio - video data based on a set scoring rule to obtain a scoring result; A display module, configured to generate and display a test report based on the second audio - video data, the scoring result, and the test case test result.

10. An electronic device, characterized in that, Comprising: A memory and one or more processors; The memory is configured to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the automated test method for a video conferencing system according to any one of claims 1 - 7.

Images