A method for testing accuracy of AI scene recognition

Abstract

The application relates to a test technology of an AI scene, and discloses a method for automatically testing AI scene recognition accuracy, which solves the problems of large manpower consumption, low test efficiency and low scene coverage rate in the scheme for testing AI scene recognition rate in traditional technologies. The scheme first initializes parameter settings, and then automatically executes a test process, including the following steps: 1. reading the type and quantity of the initialized film source, accessing a media asset library, obtaining film information and dumping the film information to a database; 2. accessing the dumped data, reading the film source data and the corresponding license party in the data, and generating a command that can be directly executed by a terminal; 3. sending an adb command to a television to play a video; 4. judging whether the video is successfully started, if the video is still failed to be started after timeout, recording the number of times of starting failure; 5. judging a scene recognition result, if the result is consistent, the scene recognition is successful, and if not, the scene recognition fails; 6. repeating steps 2 to 5 until all film sources of the current type are tested; and 7. repeating steps 1 to 6 until all preset types are tested.

Description

TECHNICAL FIELD

[0001] The present application relates to the test technology of AI scene, in particular to a method for testing the recognition accuracy of AI scene. BACKGROUND

[0002] With the development of artificial intelligence, in the traditional TV audio-visual field, AI scene recognition algorithm is accessed, so that users can intelligently recognize the current playing scene when watching video programs, and automatically switch to the image mode and sound mode suitable for watching, expecting to achieve better audio-visual effect and improve user experience. At this time, the recognition accuracy is particularly important, which directly determines the influence of the introduction of the technology on user experience.

[0003] At present, the test of scene recognition adopts manual test. The specific scheme is to find a plurality of film sources of various types such as movies, TV series, variety shows and children's programs by manual searching, then click to play, and observe the scene recognition result, so as to judge whether the recognition is correct or incorrect. In order to obtain accurate test results, hundreds of film sources need to be tested for each scene in each round, which consumes a lot of manpower, has low test efficiency, and the scene coverage rate cannot be guaranteed. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a method for testing the recognition accuracy of AI scene, which solves the problems of consuming a lot of manpower, low test efficiency and low scene coverage rate in the prior art.

[0005] The technical scheme adopted by the present application to solve the above technical problem is:

[0006] A method for testing the recognition accuracy of AI scene, applied to a computer for executing test, comprising the following steps of:

[0007] S1, accessing a media asset library to obtain a set number of film information under a corresponding film source type, and storing the film information in a database;

[0008] S2, reading the stored film information from the database to generate a command case for starting a license plate side video;

[0009] S3, sending the command case for starting to the television to execute video playing;

[0010] S4, reading the television playing log to determine whether the starting is successful, if yes, entering step S5, otherwise, recording the current film source type video playing failure +1, and entering step S6;

[0011] S5, read the scene clip source log recorded by the television scene recognition algorithm, judge whether the scene recognition is correct, if yes, record the number of correct scene recognition of the current clip source type +1, enter step S6, otherwise, record the number of incorrect scene recognition of the current clip source type +1, enter step S6;

[0012] S6, judge whether the current video is the last video of the type, if yes, store the play record result and the recognition result to the test result file, enter step S7, otherwise, return to step S2;

[0013] S7, judge whether all types of clip sources are tested, if yes, output the test result file and end the process; otherwise, return to step S1.

[0014] Further, the initialization test data includes: setting the clip source type to be tested, the test number of each type of clip source, and the timeout duration in the test process.

[0015] Further, in step S1, the accessing the media library specifically includes: accessing the media library mongodb through the pymongo module.

[0016] Further, in step S1, the obtaining the set number of film information under the corresponding clip source type specifically includes: selecting a clip source type from the set clip source type to be tested, and obtaining the set test number of film information of the clip source type from the media library.

[0017] Further, in step S1, the dumping to the database specifically includes: using the sqlite3 module to dump the obtained film information to the media_info table of sqlite.

[0018] Further, in step S2, the reading the dumped film information by accessing the database, generating a command case for starting the play of the license plate side video, specifically includes:

[0019] Querying the cover_id, video_id and corresponding license plate side playtype required for starting the play of the video from sqlite, and combining them into an executable terminal application start command.

[0020] Further, in step S3, the sending the start command case to the television specifically includes: sending the start command case to the television through the os module (operating system interface module) of python.

[0021] Further, in step S4, the reading the television play log specifically includes: reading the television play log through the subprocess module of python.

[0022] Further, in step S4, the television play log is read, if it is judged that the starting fails, and reaches the timeout length set in the initialization test data, still cannot normally start, then record the current video play failure +1 of the type of the source.

[0023] Further, in step S5, the judgment of whether the scene recognition is correct specifically includes: judging whether the scene type of the current video in the scene source log recorded by the television scene recognition algorithm is consistent with the type of the source currently tested, if consistent, then judging that the scene recognition is correct, otherwise judging that the scene recognition is incorrect.

[0024] The beneficial effects of the present application are:

[0025] The present application automatically obtains various types of test sources from the database, generates and runs scripts for starting each type of online film from the mainstream network video APP, compares the scene recognition result log to judge the recognition result, can complete the AI scene recognition accuracy test without manual intervention, greatly improves the test efficiency and test base, reduces the test manpower and time cost, shortens the test period, and accelerates the product listing. It can also obtain the latest popular sources in real time for daily tracking and verification, and solve potential problems in advance. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 The method flowchart for automatically testing the AI scene recognition accuracy in the embodiments of the present application. DETAILED DESCRIPTION

[0027] The present application aims to provide a method for automatically testing the AI scene recognition accuracy, to solve the problems of consuming a large amount of manpower, low test efficiency, and low scene coverage rate in the traditional AI scene recognition test scheme.

[0028] The scheme first initializes the parameter settings, such as the type of the source to be tested, the number of tests for each type of source, and the timeout length during the test process, and then automatically executes the test process, mainly including:

[0029] Step 1, read the initialization settings of the type and number of sources, access the media mongodb through the pymongo module, obtain real-time film / TV show / variety show / children's type film information, and use the sqlite3(database access module) module to dump to the media_info(media information) table in the sqlite(embedded relational database).

[0030] Step 2, access the data dumped in step 1, read the source data therein, including cover_id and video_id, and the corresponding license party, and combine to generate a command that can be directly executed by the terminal.

[0031] Step 3, send the adb command generated in step 2 to the TV through the os module (operating system interface module) of python, and play the video.

[0032] Step 4, read the log through the subprocess module (system command execution module) of python to determine whether the start is successful, if it is still unsuccessful after timeout, record start failure + 1.

[0033] Step 5, read the log through the subprocess module of python to determine the scene recognition result, compare the source type filtered in step 2, consistent scene recognition success + 1, inconsistent or timeout not obtained, scene recognition failure + 1.

[0034] Step 6, repeat steps 2 to 5 until all the source tests under the current type are completed, then record the test results to the result file.

[0035] Step 7, repeat steps 1 to 6 until all the preset types are tested, complete the test.

[0036] Embodiment:

[0037] The above scheme of the present application will be described in detail in combination with the technical implementation process of the accompanying drawings. Figure 1 The present embodiment uses python language to develop test script.

[0038] Before testing, the computer connected through adb needs to be executed and the TV to be tested.

[0039] Step 1, set the initialization data, including the types of source to be tested types (eg: movie / tv / variety / cartoon……), the number of times of each source test times (eg: 200 times), and the time of timeout time (eg: 20 seconds).

[0040] Step 2, access the mongodb of media through the pymongo module, according to the types and times set in step 1, find the corresponding resources from the database, such as finding 200 movie, db.getCollection('tables').find({"Model":"movie"}).limit(200), and using sqlite3 module to dump to the media_info table of sqlite.

[0041] Step 3, query the cover_id, video_id and corresponding license playtype required for video start from sqlite, and refer to the command SELECT cover_id, video_id, playtype FROM media_info WHERE Model="movie", and combine into executable terminal application start command, such as playtype=tencent, the instruction is: adb shell am start com.ktcp.video-acom.tencent.qqlivetv.open-dtenvideo2: / / ?action=7&cover_id="cover_id"&video_id="video_id"

[0042] Step 4, execute the terminal application start command generated in step 3 through the os module of python, such as: os.system("adb command")

[0043] Step 5, find whether there is a new isPlayingtrue in the log through the subprocess module of python, and judge whether it is correctly started. Such as: subprocess.Popen("cat / data / logcat.txt\"|grep'isPlaybing true'\"", shell=True), if the timeout time set in step 1 is reached, record the start failure, and continue to play the next video.

[0044] Step 6, after successful start, use the same method in step 5 to obtain the current scene source log recorded by the scene recognition algorithm, and judge whether it is consistent with the type "movie" queried in step 3, if consistent, pass+1, if inconsistent or the log is not obtained after reaching the timeout time, fail+1.

[0045] Step 7, repeat steps 3 to 6 until 200 video sources of the current video source type are tested, and record the test results of pass and fail to the result file.

[0046] Step 8, repeat steps 2 to 7 to test 200 video sources of the next video source type until all video source types are tested.

[0047] Finally, it should be pointed out that the above embodiments are only preferred embodiments and do not limit the present application. It should be noted that for those skilled in the art, without departing from the purpose of the present application and the scope of the claims, a number of modifications, equivalent replacements, improvements, etc. can be made, which should be included in the protection scope of the present application.

Claims

1. A method for automatically testing the accuracy of AI scene recognition, applied to a computer performing the test, characterized in that: The method includes: connecting a computer used to perform the test and a television set to be tested via ADB, initializing the test data, and automatically performing the following steps: S1. Access the media resource library, obtain a set number of film information under the corresponding source type, and dump it into the database; S2. Access the database to read the dumped film information and generate a command use case for the license holder to start playing the video; S3. Send a start command to the TV to play the video. S4. Read the TV play log to determine whether the start of the broadcast is successful. If so, proceed to step S5. Otherwise, record the current source type video playback failure + 1, and proceed to step S6. S5. Read the scene source log recorded by the TV scene recognition algorithm to determine whether the scene recognition is correct. If correct, record the number of correct scene recognitions of the current source type + 1, and proceed to step S6. Otherwise, record the number of incorrect scene recognitions of the current source type + 1, and proceed to step S6. Determining whether the scene recognition is correct specifically includes: determining whether the scene type of the current video in the scene source log recorded by the TV scene recognition algorithm is consistent with the type of the currently tested source. If they are consistent, determining that the scene recognition is correct; otherwise, determining that the scene recognition is incorrect. S6. Determine whether the current video is the last video of this type. If so, store the playback record result and the recognition result in the test result file and proceed to step S7. Otherwise, return to step S2. S7. Determine whether all types of film sources have been tested. If so, output the test result file and end the process; otherwise, return to step S1.

2. The method for automatically testing AI scene recognition accuracy according to claim 1, characterized in that: The initialization test data includes: setting the type of film source to be tested, the number of tests for each type of film source, and the timeout duration during the test process.

3. The method for automatically testing AI scene recognition accuracy according to claim 1, characterized in that: In step S1, accessing the media resource library specifically includes: accessing the media resource library mongodb through the pymongo module.

4. The method for automatically testing AI scene recognition accuracy according to claim 2, characterized in that: In step S1, obtaining a set number of film information of a corresponding film source type specifically includes: Select a source type from the set source types to be tested, and obtain the set test quantity of film information under the source type from the media resource library.

5. The method for automatically testing AI scene recognition accuracy according to claim 1, characterized in that: In step S1, dumping to the database specifically includes: using the sqlite3 module to dump the obtained movie information into the media_info table of sqlite.

6. The method for automatically testing AI scene recognition accuracy according to claim 1, characterized in that: In step S2, the method of accessing the database, reading the dumped film information, and generating a command use case for the licensee to start playing the video specifically includes: Query the cover_id, video_id, and corresponding license holder playtype required to start the video from SQLite, and combine them into an executable terminal application start command.

7. The method for automatically testing AI scene recognition accuracy according to claim 6, characterized in that: In step S3, the sending of the start broadcast command example to the television specifically includes: sending the start broadcast command example to the television through the Python os module.

8. The method for automatically testing AI scene recognition accuracy according to claim 1, characterized in that: In step S4, the reading of the TV play log specifically includes: reading the TV play log through the subprocess module of Python.

9. The method for automatically testing AI scene recognition accuracy according to claim 1, characterized in that: In step S4, the TV play log is read. If it is determined that the start of the broadcast fails and the timeout period set in the initialization test data is reached and the broadcast still cannot be started normally, the current source type video playback failure + 1 is recorded.

Images