Audio and video test method and system for intelligent terminal

Abstract

The invention provides an audio and video test method and system for an intelligent terminal. The method comprises the following steps: acquiring video data; wherein the video data comprises video data and audio data; identifying a tooth image, a face image, a tongue image and a throat image from the video data; extracting audio data and subtitle data corresponding to the tooth image, the face image, the tongue image and the throat image; converting the tooth image, the face image, the tongue image and the throat image into a tooth motion sequence, a face motion sequence, a tongue motion sequence and a throat motion sequence; creating test class objects of the audio data and the subtitle data, determining code positions of the test class objects of the audio data and the subtitle data, and generating a test API interface according to the code positions; and inputting the tooth motion sequence, the face motion sequence, the tongue motion sequence, the throat motion sequence, the audio data and the subtitle data into the test API interface for testing to obtain the specific occupancy rate and the power consumption change data of the intelligent terminal, and establishing a new audio and video linkage test class. Two APIs need to be packaged separately for different audios or videos, code resources are saved, the testing efficiency and the automation degree are improved, the labor cost is effectively reduced, the deviation degree of video images and audios or subtitles is calibrated through data predicted by an intelligent model, and a more accurate testing result is provided.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and particularly to an audio - video test system for an intelligent terminal, an audio - video test method for an intelligent terminal, an intelligent terminal, and a storage medium. Background Art

[0002] Mobile terminal audio - video testing is a key step to ensure the normal operation and performance compliance of audio and video functions on mobile devices (such as mobile phones, tablets). Usually, automated testing tools can be used for testing, such as Appium, Espresso (Android), XCTest (iOS). On the other hand, performance monitoring tools can also be used for testing, such as Android Profiler, Xcode Instruments. In addition, audio - video analysis tools can be used for testing, such as Audacity, FFmpeg, VLC. In terms of video testing, cross - platform test scripts are written using Appium + Python / Java, automated test cases are designed for core functions (playback control, synchronization), and AI image recognition (such as OpenCV) is used to automatically detect video image quality problems. Audio and video are tested separately, and how to perform audio - video linkage testing is an urgent problem to be solved. Summary of the Invention

[0003] In view of the above problems, embodiments of the present invention are proposed to provide an audio - video test method for an intelligent terminal, an audio - video test system for an intelligent terminal, an audio - video test system for an intelligent terminal, an intelligent terminal, and a storage medium that overcome the above problems or at least partially solve the above problems.

[0004] To solve the above problems, embodiments of the present invention disclose an audio - video test method for an intelligent terminal, including:

[0005] Obtaining video - recording data; wherein, the video - recording data includes video data and audio data;

[0006] Identifying tooth images, face images, tongue images, and throat images from the video data;

[0007] Extracting the audio data and subtitle data corresponding to the tooth images, face images, tongue images, and throat images;

[0008] Converting the tooth images, face images, tongue images, and throat images into tooth movement sequences, face movement sequences, tongue movement sequences, and throat movement sequences;

[0009] Create test class objects for the audio data and subtitle data and determine the code locations of the test class objects for the audio data and subtitle data, and generate a test API interface based on the code locations;

[0010] Input the tooth movement sequence, facial movement sequence, tongue movement sequence, laryngeal movement sequence, audio data, and subtitle data into the test API interface for testing to obtain the specific occupancy rate and power consumption change data of the intelligent terminal.

[0011] Preferably, the method further includes:

[0012] Input the tooth movement sequence, facial movement sequence, tongue movement sequence, laryngeal movement sequence, audio data, and subtitle data into the intelligent recognition model to obtain the output first matching prediction coefficient.

[0013] Preferably, the identifying the tooth image, facial image, tongue image, and laryngeal image from the video data includes:

[0014] Decompose the video data into multiple frame images;

[0015] Traverse the multiple frame images and divide the multiple frame images into portrait images and non-portrait images;

[0016] Identify the face region of the portrait image;

[0017] Extract texture features for the face region to obtain the tooth image, facial image, tongue image, and laryngeal image.

[0018] Preferably, the inputting the tooth movement sequence, facial movement sequence, tongue movement sequence, laryngeal movement sequence, audio data, and subtitle data into the test API interface for testing to obtain the specific occupancy rate and power consumption change data of the intelligent terminal includes:

[0019] Input the tooth movement sequence, facial movement sequence, tongue movement sequence, laryngeal movement sequence, audio data, and subtitle data into the test API interface, and calculate to obtain a second matching prediction coefficient;

[0020] Adjust the second matching prediction coefficient according to the first matching prediction coefficient to obtain an adjusted matching prediction coefficient;

[0021] Correct the matching degrees of the tooth movement sequence, facial movement sequence, tongue movement sequence, laryngeal movement sequence, audio data, and subtitle data according to the adjusted matching prediction coefficient, and then conduct testing to obtain the CPU occupancy rate, GPU occupancy rate, memory occupancy rate, and power consumption change data of the intelligent terminal.

[0022] Preferably, inputting the tooth movement sequence, facial movement sequence, tongue movement sequence, laryngeal movement sequence, audio data, and subtitle data into the intelligent recognition model to obtain the output first matching prediction coefficient includes:

[0023] Converting the tooth movement sequence, facial movement sequence, tongue movement sequence, laryngeal movement sequence, audio data, and subtitle data into a first feature vector through a feature extraction framework;

[0024] Performing downsampling and convolutional operator transformation on the first feature vector to obtain a second feature vector;

[0025] Fusing the second feature vectors in a way of channel dimension connection to obtain a fused feature;

[0026] Constructing multiple neighbor nodes based on the fused feature to obtain a k-NN local graph;

[0027] Performing a neighborhood aggregation operation on the edge features of the k-NN local graph to obtain an updated feature;

[0028] Inputting the updated feature into a binary classification module to obtain a trained intelligent recognition model, and inputting new tooth movement sequences, facial movement sequences, tongue movement sequences, laryngeal movement sequences, audio data, and subtitle data into the trained intelligent recognition model to output the first matching prediction coefficient.

[0029] Preferably, adjusting the second matching prediction coefficient according to the first matching prediction coefficient to obtain an adjusted matching prediction coefficient includes:

[0030] Converting and generating an audio track subtitle adjustment amplitude according to the first matching prediction coefficient;

[0031] Adding the audio track subtitle adjustment amplitude to the second matching prediction coefficient to obtain an adjusted matching prediction coefficient.

[0032] An embodiment of the present invention discloses an audio-visual test system for an intelligent terminal, including:

[0033] A video recording data acquisition module for acquiring video recording data; wherein, the video recording data includes video data and audio data;

[0034] An identification module for identifying tooth images, facial images, tongue images, and laryngeal images from the video data;

[0035] An extraction module for extracting audio data and subtitle data corresponding to the tooth images, facial images, tongue images, and laryngeal images;

[0036] A conversion module for converting the tooth image, face image, tongue image, and larynx image into a tooth movement sequence, a face movement sequence, a tongue movement sequence, and a larynx movement sequence;

[0037] A determination module for creating test class objects of the audio data and subtitle data and determining the code positions of the test class objects of the audio data and subtitle data, and generating a test API interface according to the code positions;

[0038] A test module for inputting the tooth movement sequence, face movement sequence, tongue movement sequence, larynx movement sequence, audio data, and subtitle data into the test API interface for testing to obtain specific occupancy rate and power consumption change data of the intelligent terminal.

[0039] Preferably, the system further includes:

[0040] An output module for inputting the tooth movement sequence, face movement sequence, tongue movement sequence, larynx movement sequence, audio data, and subtitle data into an intelligent recognition model to obtain an output first matching prediction coefficient.

[0041] An embodiment of the present invention also discloses an intelligent terminal, including a memory and a processor, where the memory stores a computer program, and the processor implements the steps of the audio-video test of the intelligent terminal as described above when executing the computer program.

[0042] An embodiment of the present invention also discloses a computer-readable storage medium, on which a computer program is stored, and the computer program implements the steps of the audio-video test of the intelligent terminal as described above when executed by a processor.

[0043] The embodiments of the present invention have the following advantages:

[0044] In an embodiment of the present invention, the audio - video testing method of the intelligent terminal includes: obtaining video - recording data; wherein, the video - recording data includes video data and audio data; identifying tooth images, face images, tongue images, and laryngeal images from the video data; extracting the audio data and subtitle data corresponding to the tooth images, face images, tongue images, and laryngeal images; converting the tooth images, face images, tongue images, and laryngeal images into tooth movement sequences, face movement sequences, tongue movement sequences, and laryngeal movement sequences; creating test class objects for the audio data and subtitle data and determining the code positions of the test class objects for the audio data and subtitle data, generating test API interfaces according to the code positions; inputting the tooth movement sequences, face movement sequences, tongue movement sequences, and laryngeal movement sequences, as well as the audio data and subtitle data, into the test API interfaces for testing, obtaining the specific occupancy rate and power consumption change data of the intelligent terminal, establishing a new audio - video linkage test class. Different audio or videos need to separately encapsulate two API interfaces, saving code resources, improving the test efficiency and automation level, effectively reducing labor costs, and using the data predicted by the intelligent model to calibrate the deviation between the video image and the audio or subtitle, providing more accurate test results. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0046] Figure 1 It is a flowchart of the steps of an embodiment of the audio - video testing method of an intelligent terminal according to an embodiment of the present invention;

[0047] Figure 2 It is a block diagram of the structure of an embodiment of the audio - video testing system of an intelligent terminal according to an embodiment of the present invention;

[0048] Figure 3 It is an internal structure diagram of an intelligent terminal in an embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0049] In order to make the technical problems, technical solutions, and beneficial effects solved by the embodiments of the present invention more clearly understood, the following further details the embodiments of the present invention in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0050] In an embodiment of the present invention, the images in the video data and the audio data are subjected to image and audio data processing to obtain a regional image of the parts related to the human vocal movement in the image and the corresponding audio data. Then, a new audio-visual linkage test class object is generated, and the new test class object tests the image of the parts related to the human vocal movement and the corresponding audio data to obtain the test information of the terminal. On the other hand, calibration operations are also performed on the image of the parts related to the human vocal movement and the corresponding audio data through an intelligent model, improving the accuracy of the audio-visual test samples.

[0051] The audio-visual test method for an intelligent terminal provided by an embodiment of the present invention can be applied to an application environment including an intelligent terminal and a server. Among them, the intelligent terminal communicates with the server through a network. The method can also be applied to various different intelligent terminals such as personal computers, laptop computers, smart phones, and portable wearable devices. The embodiment of the present invention does not limit the specific type of the intelligent terminal. The server can be implemented by an independent server or a server cluster composed of multiple servers. The embodiment of the present invention does not impose too many restrictions on this.

[0052] Refer to Figure 1 , which shows the step flowchart of an embodiment of the audio-visual test method for an intelligent terminal according to an embodiment of the present invention. Specifically, it may include the following steps:

[0053] Step 101, obtain video data; wherein, the video data includes video data and audio data;

[0054] In an embodiment of the present invention, the video data may include audio-visual data. Specifically, the video data may include image data and subtitle data, and the subtitle data has a corresponding relationship with the audio data. In an embodiment of the present invention, the video data, subtitle data, and audio data of the video data for testing need to be adjusted in the following manner.

[0055] Step 102, identify the tooth image, face image, tongue image, and larynx image from the video data;

[0056] First, in an embodiment of the present invention, the image data obtained by converting the video data needs to be processed, and then the regional image of the parts related to the human vocal movement in each frame of the image data is identified. Specifically as follows: The identifying the tooth image, face image, tongue image, and larynx image from the video data includes: decomposing the video data into multiple frame images; traversing the multiple frame images, dividing the multiple frame images into portrait images and non-portrait images; identifying the face area of the portrait image; and extracting texture features for the face area to obtain the tooth image, face image, tongue image, and larynx image.

[0057] Specifically, first, it can be identified whether each frame of image contains a portrait image. If the frame of image contains a portrait image, it is classified into the portrait image set. The facial region of the portrait image is identified from the portrait image set, and then through the method of texture feature extraction, the dental image, facial image, tongue image, and laryngeal image of the facial region of the portrait image are identified. Specifically applied to the embodiments of the present invention, the regional image of the human body part can be extracted by local binary pattern to obtain a preliminary regional image of the human body part, and then the features of the preliminary regional image of the human body part are identified by scale-invariant feature transform method to obtain the final result, which meets the needs of the image size, improves the range of feature recognition, and improves the accuracy of feature recognition.

[0058] Step 103: Extract the audio data and subtitle data corresponding to the dental image, facial image, tongue image, and laryngeal image.

[0059] Further applied to the embodiments of the present invention, the audio data and subtitle data corresponding to the dental image, facial image, tongue image, and laryngeal image can be extracted. In the embodiments of the present invention, the subtitle data and the audio data have a first corresponding relationship, and the video data has a second corresponding relationship with any one of the subtitle data or the audio data. Then, the extracted dental image, facial image, tongue image, and laryngeal image have a corresponding relationship with any one of the audio data and the subtitle data. From the perspective of the image, it is to associate the time stamp of each frame including the portrait image with the audio track time stamp of the audio data, that is, the audio data and subtitle data corresponding to the dental image, facial image, tongue image, and laryngeal image can be extracted.

[0060] Step 104: Convert the dental image, facial image, tongue image, and laryngeal image into a dental motion sequence, a facial motion sequence, a tongue motion sequence, and a laryngeal motion sequence.

[0061] Further applied to the embodiments of the present invention, the intelligent terminal can also convert the dental image, facial image, tongue image, and laryngeal image into a dental motion sequence, a facial motion sequence, a tongue motion sequence, and a laryngeal motion sequence. Specifically, the pixel coordinate combinations of the dental image, facial image, tongue image, and laryngeal image can be converted into corresponding dental motion sequence, facial motion sequence, tongue motion sequence, and laryngeal motion sequence. For example, the pixel coordinates can be RGB pixel coordinates or YUV pixel coordinates. For example, if the RGB pixel coordinates of a certain dental image are as follows (the values in the brackets represent the gray values composed of the RGB components of the pixel): (255)(70)(169); (245)(120)(0); (255)(70)(169);

[0065] Then the tooth movement sequence corresponding to the converted tooth image is 25570169245120025570169; The above conversion process is only an example of the embodiments of the present invention. The tooth movement sequence corresponding to the tooth image can also be obtained through other conversion methods, and the embodiments of the present invention do not impose too many restrictions on this.

[0066] Step 105: Create test class objects for the audio data and subtitle data and determine the code positions of the test class objects for the audio data and subtitle data, and generate a test API interface according to the code positions.

[0067] Further applied to the embodiments of the present invention, after obtaining the movement sequence, test class objects for the audio data and subtitle data can be created and the code positions of the test class objects for the audio data and subtitle data can be determined, and a test API interface can be generated according to the code positions, that is, a specific test API interface is generated according to the encapsulation of relevant codes.

[0068] In the embodiments of the present invention, a new test class object is used to test the performance and power consumption of the terminal hardware for audio-video linkage testing, and a new audio-video linkage test class is established. Two API interfaces need to be separately encapsulated for different audio or videos, which saves code resources, improves the test efficiency and automation level, and effectively reduces the labor cost.

[0069] Step 106: Input the tooth movement sequence, facial movement sequence, tongue movement sequence, laryngeal movement sequence, audio data, and subtitle data into the test API interface for testing to obtain the specific occupancy rate and power consumption change data of the intelligent terminal.

[0070] In the embodiments of the present invention, the test API interface is called, and the tooth movement sequence, facial movement sequence, tongue movement sequence, laryngeal movement sequence, audio data, and subtitle data are input into the test API interface for testing to obtain the specific occupancy rate and power consumption change data of the intelligent terminal. The function of the test API interface is to measure the deviation degree, hardware performance, and power consumption performance of the two according to the above sequence data, audio data, and subtitle data. Specifically, the specific occupancy rate may include CPU occupancy rate, GPU occupancy rate, memory occupancy rate, bit rate, frame rate, etc., and the embodiments of the present invention do not impose too many restrictions on this.

[0071] Specifically in the embodiments of the present invention, the step of inputting the tooth movement sequence, facial movement sequence, tongue movement sequence, laryngeal movement sequence, audio data, and subtitle data into the test API interface for testing to obtain the specific occupancy rate and power consumption change data of the intelligent terminal includes:

[0072] Input the tooth movement sequence, facial movement sequence, tongue movement sequence, laryngeal movement sequence, audio data, and subtitle data into the test API interface to calculate the second matching prediction coefficient;

[0073] Adjust the second matching prediction coefficient according to the first matching prediction coefficient to obtain an adjusted matching prediction coefficient;

[0074] According to the adjusted matching prediction coefficient, correct the matching degree of the tooth movement sequence, facial movement sequence, tongue movement sequence, laryngeal movement sequence, audio data, and subtitle data, and then conduct a test to obtain the CPU occupancy rate, GPU occupancy rate, memory occupancy rate, and power consumption change data of the smart terminal.

[0075] Specifically, the second matching prediction coefficient refers to the deviation degree between the video image and the audio or subtitle, that is, the time difference between the video image and the audio or subtitle;

[0076] The first matching prediction coefficient can be the time difference between the video image and the audio or subtitle predicted by the intelligent model;

[0077] The CPU occupancy rate refers to the ratio of the number of CPU threads occupied during measurement to the total number of threads; the GPU occupancy rate refers to the ratio of the number of GPU threads occupied during measurement to the total number of threads; the memory occupancy rate refers to the ratio of the amount of memory consumed by the test process to the total amount of memory; the power consumption change data refers to the degree of change in power consumption during the test process to the total power;

[0078] Specifically, the adjusting the second matching prediction coefficient according to the first matching prediction coefficient to obtain an adjusted matching prediction coefficient includes: generating an audio track subtitle adjustment amplitude according to the first matching prediction coefficient; adding the audio track subtitle adjustment amplitude to the second matching prediction coefficient to obtain an adjusted matching prediction coefficient.

[0079] For example, the audio track subtitle adjustment amplitude can be that the audio track moves forward by 0.21 seconds or the audio track moves backward by 0.13 seconds; then combine the audio track subtitle adjustment amplitude with the second matching prediction coefficient to obtain an adjusted matching prediction coefficient. In the embodiments of the present invention, the data predicted by the intelligent model is used to calibrate the deviation degree between the video image and the audio or subtitle, providing a more accurate test result.

[0080] Specifically, in the embodiments of the present invention, the first matching prediction coefficient can be obtained in the following manner: input the tooth movement sequence, facial movement sequence, tongue movement sequence, laryngeal movement sequence, audio data, and subtitle data into the intelligent recognition model to obtain the output first matching prediction coefficient.

[0081] Specifically applied to the embodiments of the present invention, the input of the tooth movement sequence, facial movement sequence, tongue movement sequence, laryngeal movement sequence, audio data, and subtitle data into the intelligent recognition model to obtain the output first matching prediction coefficient includes:

[0082] Converting the tooth movement sequence, facial movement sequence, tongue movement sequence, laryngeal movement sequence, audio data, and subtitle data into a first feature vector through a feature extraction framework;

[0083] Performing downsampling and convolution operator transformation on the first feature vector to obtain a second feature vector;

[0084] Fusing the second feature vector by means of channel dimension connection to obtain a fused feature;

[0085] Constructing multiple neighbor nodes based on the fused feature to obtain a k-NN local graph;

[0086] Performing a neighborhood aggregation operation on the edge features of the k-NN local graph to obtain an updated feature;

[0087] Inputting the updated feature into a binary classification module to obtain a trained intelligent recognition model, and inputting new tooth movement sequences, facial movement sequences, tongue movement sequences, laryngeal movement sequences, audio data, and subtitle data into the trained intelligent recognition model to output the first matching prediction coefficient.

[0088] In the embodiments of the present invention, the tooth movement sequence, facial movement sequence, tongue movement sequence, laryngeal movement sequence, audio data, and subtitle data are used as sample data, and the matching prediction coefficient corresponding to the sample data is used as label data. It should be noted that the matching prediction coefficient of the sample data can be obtained by manual marking. The intelligent model may include a feature extraction framework module, a downsampling and convolution operator transformation module, a k-NN local graph module, a neighborhood aggregation operation module, a binary classification module, etc. Of course, it may also include other modules, and the embodiments of the present invention do not limit this too much.

[0089] Specifically, the feature extraction framework module may include sub-modules such as temporal scale feature extraction. The first feature vector can be expressed as:

[0090] Xt = {Xai ∈ D1, Xbi ∈ D2, Xci ∈ D3, Xdi ∈ D4}; where Xai represents the data of the tooth movement sequence, Xbi represents the data of the facial movement sequence, Xci represents the data of the tongue movement sequence, Xdi represents the data of the laryngeal movement sequence, i = 1, 2, 3, ······, n, where n is a positive integer, D1 represents the feature dimension of the tooth image, D2 represents the feature dimension of the facial image, D3 represents the feature dimension of the tongue image, and D4 represents the feature dimension of the laryngeal image;

[0091] Further, the downsampling and convolutional operator transformation method is expressed as follows, that is, the second feature vector is Xa:

[0092] Xa = Con1D(AvgPool(Con1D(Xt))), a = (1, 2, 3, 4);

[0093] Then the fused feature Xr is expressed as follows:

[0094] Xr = Concat(X1, X2, X3, X4);

[0095] It should be noted that the way to obtain the k-NN local graph can be through the extended k-NN algorithm. Multiple neighbor nodes are constructed according to the fused feature to obtain the k-NN local graph. Neighborhood aggregation operations are performed on the edge features of the k-NN local graph to obtain updated features; the updated features are input into a binary classification module to complete the training of the intelligent recognition model. In the preferred embodiment of the present invention, this binary classification module can be a classification and regression model such as a support vector machine module. The present invention embodiment does not limit this too much.

[0096] In the embodiment of the present invention, the intelligent model is also used to identify the tooth movement sequence, facial movement sequence, tongue movement sequence, laryngeal movement sequence, audio data, and subtitle data to obtain the predicted first matching prediction coefficient.

[0097] In the embodiment of the present invention, the audio-visual test method of the intelligent terminal includes: obtaining video data; wherein, the video data includes video data and audio data; identifying tooth images, facial images, tongue images, and laryngeal images from the video data; extracting the audio data and subtitle data corresponding to the tooth images, facial images, tongue images, and laryngeal images; converting the tooth images, facial images, tongue images, and laryngeal images into a tooth movement sequence, facial movement sequence, tongue movement sequence, and laryngeal movement sequence; creating test class objects for the audio data and subtitle data and determining the code positions of the test class objects for the audio data and subtitle data, generating test API interfaces according to the code positions; inputting the tooth movement sequence, facial movement sequence, tongue movement sequence, laryngeal movement sequence, audio data, and subtitle data into the test API interfaces for testing to obtain the specific occupancy rate and power consumption change data of the intelligent terminal, establishing a new audio-visual linkage test class. Different audio or videos need to separately package two API interfaces, saving code resources, improving the test efficiency and automation degree, effectively reducing the labor cost, and using the data predicted by the intelligent model to calibrate the deviation between the video image and the audio or subtitle, providing more accurate test results.

[0098] It should be noted that, for the method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that the embodiments of the present invention are not limited by the described action sequences, because according to the embodiments of the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily essential for the embodiments of the present invention.

[0099] Referring to Figure 2 , a structural block diagram of an audio - video test system embodiment of an intelligent terminal according to an embodiment of the present invention is shown. The intelligent terminal includes a display area, and the intelligent terminal runs multiple application programs. Specifically, it may include the following modules:

[0100] The video recording data acquisition module 301 is used to acquire video recording data. Among them, the video recording data includes video data and audio data.

[0101] The recognition module 302 is used to recognize a tooth image, a face image, a tongue image, and a throat image from the video data.

[0102] The extraction module 303 is used to extract the audio data and subtitle data corresponding to the tooth image, the face image, the tongue image, and the throat image.

[0103] The conversion module 304 is used to convert the tooth image, the face image, the tongue image, and the throat image into a tooth movement sequence, a face movement sequence, a tongue movement sequence, and a throat movement sequence.

[0104] The determination module 305 is used to create test class objects of the audio data and subtitle data and determine the code positions of the test class objects of the audio data and subtitle data, and generate test API interfaces according to the code positions.

[0105] The test module 306 is used to input the tooth movement sequence, the face movement sequence, the tongue movement sequence, the throat movement sequence, the audio data, and the subtitle data into the test API interface for testing, and obtain the specific occupancy rate and power consumption change data of the intelligent terminal.

[0106] Preferably, the system further includes:

[0107] The output module is used to input the tooth movement sequence, the face movement sequence, the tongue movement sequence, the throat movement sequence, the audio data, and the subtitle data into an intelligent recognition model to obtain an output first matching prediction coefficient.

[0108] Preferably, the recognition module includes:

[0109] A decomposition sub-module, configured to decompose the video data into multiple frame images;

[0110] A traversal sub-module, configured to traverse the multiple frame images and divide the multiple frame images into portrait images and non-portrait images;

[0111] An identification sub-module, configured to identify the facial region of the portrait image;

[0112] An extraction sub-module, configured to extract texture features for the facial region to obtain dental images, facial images, tongue images, and laryngeal images.

[0113] Preferably, the test module includes:

[0114] An input sub-module, configured to input the dental motion sequence, facial motion sequence, tongue motion sequence, laryngeal motion sequence, audio data, and subtitle data into the test API interface to calculate a second matching prediction coefficient;

[0115] An adjustment sub-module, configured to adjust the second matching prediction coefficient according to the first matching prediction coefficient to obtain an adjusted matching prediction coefficient;

[0116] A corrected test sub-module, configured to correct the matching degrees of the dental motion sequence, facial motion sequence, tongue motion sequence, laryngeal motion sequence, audio data, and subtitle data according to the adjusted matching prediction coefficient, and then perform a test to obtain the CPU occupancy rate, GPU occupancy rate, memory occupancy rate, and power consumption change data of the intelligent terminal.

[0117] Preferably, the output module includes:

[0118] A conversion sub-module, configured to convert the dental motion sequence, facial motion sequence, tongue motion sequence, laryngeal motion sequence, audio data, and subtitle data into a first feature vector through a feature extraction framework;

[0119] A downsampling and convolutional operator transformation sub-module, configured to perform downsampling and convolutional operator transformation on the first feature vector to obtain a second feature vector;

[0120] A fusion sub-module, configured to fuse the second feature vectors by means of channel dimension connection to obtain a fused feature;

[0121] A construction sub-module, configured to construct multiple neighbor nodes according to the fused feature to obtain a k-NN local graph;

[0122] An aggregation operation sub-module, configured to perform a neighborhood aggregation operation on the edge features of the k-NN local graph to obtain an updated feature;

[0123] An output sub-module, configured to input the updated features into a binary classification module to obtain a trained intelligent recognition model, and input a new tooth movement sequence, a facial movement sequence, a tongue movement sequence, a laryngeal movement sequence, audio data, and caption data into the trained intelligent recognition model, and output a first matching prediction coefficient.

[0124] Preferably, the adjustment sub-module includes:

[0125] A generation unit, configured to convert and generate an audio track caption adjustment amplitude according to the first matching prediction coefficient;

[0126] An addition unit, configured to add the audio track caption adjustment amplitude to the second matching prediction coefficient to obtain an adjusted matching prediction coefficient.

[0127] For the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For related parts, refer to the partial description of the method embodiment.

[0128] For the specific limitations of the audio-visual test system of the intelligent terminal, reference can be made to the limitations on the audio-visual test method of the intelligent terminal in the above text, which will not be elaborated here. Each module in the above audio-visual test system of the intelligent terminal can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor of the intelligent terminal in hardware form or independent of it, or stored in the memory of the intelligent terminal in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0129] The above-provided audio-visual test system of the intelligent terminal can be used to execute the audio-visual test method of the intelligent terminal provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0130] In one embodiment, an intelligent terminal is provided, and its internal structure diagram can be as Figure 3 shown. The intelligent terminal includes a processor, a memory, a network interface, a display screen, and an input system connected through a system bus. Among them, the processor of the intelligent terminal is used to provide computing and control capabilities. The memory of the intelligent terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the intelligent terminal is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a method for simulating the daylighting rate. The display screen of the intelligent terminal can be a liquid crystal display screen or an electronic ink display screen. The input system of the intelligent terminal can be a touch layer covered on the display screen, or a button, a trackball, or a touchpad provided on the shell of the intelligent terminal, or an external keyboard, a touchpad, or a mouse, etc.

[0131] Those skilled in the art can understand that Figure 3 the structure shown in Figure 3 is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the intelligent terminal to which the solution of this application is applied. The specific intelligent terminal may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0132] In one embodiment, an intelligent terminal is provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0133] Obtain video recording data; wherein, the video recording data includes video data and audio data;

[0134] Identify a tooth image, a face image, a tongue image, and a larynx image from the video data;

[0135] Extract the audio data and subtitle data corresponding to the tooth image, the face image, the tongue image, and the larynx image;

[0136] Convert the tooth image, the face image, the tongue image, and the larynx image into a tooth movement sequence, a face movement sequence, a tongue movement sequence, and a larynx movement sequence;

[0137] Create test class objects for the audio data and subtitle data and determine the code positions of the test class objects for the audio data and subtitle data, and generate a test API interface according to the code positions;

[0138] Input the tooth movement sequence, the face movement sequence, the tongue movement sequence, the larynx movement sequence, the audio data, and the subtitle data into the test API interface for testing to obtain the specific occupancy rate and power consumption change data of the intelligent terminal.

[0139] Preferably, the method further includes:

[0140] Input the tooth movement sequence, the face movement sequence, the tongue movement sequence, the larynx movement sequence, the audio data, and the subtitle data into an intelligent recognition model to obtain an output first matching prediction coefficient.

[0141] Preferably, the identifying a tooth image, a face image, a tongue image, and a larynx image from the video data includes:

[0142] Decompose the video data into multiple frame images;

[0143] Traverse the multiple frame images and divide the multiple frame images into portrait images and non-portrait images;

[0144] Identify the facial region of the portrait image;

[0145] Extract texture features for the facial region to obtain dental images, facial images, tongue images, and laryngeal images.

[0146] Preferably, inputting the dental motion sequence, facial motion sequence, tongue motion sequence, laryngeal motion sequence, audio data, and subtitle data into the test API interface for testing to obtain the specific occupancy rate and power consumption change data of the intelligent terminal, including:

[0147] Input the dental motion sequence, facial motion sequence, tongue motion sequence, laryngeal motion sequence, audio data, and subtitle data into the test API interface, and calculate to obtain a second matching prediction coefficient;

[0148] Adjust the second matching prediction coefficient according to the first matching prediction coefficient to obtain an adjusted matching prediction coefficient;

[0149] According to the adjusted matching prediction coefficient, correct the matching degree of the dental motion sequence, facial motion sequence, tongue motion sequence, laryngeal motion sequence, audio data, and subtitle data, and then conduct testing to obtain the CPU occupancy rate, GPU occupancy rate, memory occupancy rate, and power consumption change data of the intelligent terminal.

[0150] Preferably, inputting the dental motion sequence, facial motion sequence, tongue motion sequence, laryngeal motion sequence, audio data, and subtitle data into the intelligent recognition model to obtain the output first matching prediction coefficient, including:

[0151] Convert the dental motion sequence, facial motion sequence, tongue motion sequence, laryngeal motion sequence, audio data, and subtitle data into a first feature vector through a feature extraction framework;

[0152] Perform downsampling and convolution operator transformation on the first feature vector to obtain a second feature vector;

[0153] Fuse the second feature vectors by means of channel dimension connection to obtain a fused feature;

[0154] Construct multiple neighbor nodes according to the fused feature to obtain a k-NN local graph;

[0155] Perform neighborhood aggregation operation on the edge features of the k-NN local graph to obtain an updated feature;

[0156] Input the updated feature into a binary classification module to obtain a trained intelligent recognition model, and input new dental motion sequences, facial motion sequences, tongue motion sequences, laryngeal motion sequences, audio data, and subtitle data into the trained intelligent recognition model to output the first matching prediction coefficient.

[0157] Preferably, adjusting the second matching prediction coefficient according to the first matching prediction coefficient to obtain an adjusted matching prediction coefficient includes:

[0158] Converting and generating an audio track subtitle adjustment amplitude according to the first matching prediction coefficient;

[0159] Adding the audio track subtitle adjustment amplitude to the second matching prediction coefficient to obtain an adjusted matching prediction coefficient.

[0160] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented;

[0161] Obtaining video data; wherein the video data includes video data and audio data;

[0162] Identifying a tooth image, a face image, a tongue image, and a larynx image from the video data;

[0163] Extracting audio data and subtitle data corresponding to the tooth image, the face image, the tongue image, and the larynx image;

[0164] Converting the tooth image, the face image, the tongue image, and the larynx image into a tooth movement sequence, a face movement sequence, a tongue movement sequence, and a larynx movement sequence;

[0165] Creating a test class object for the audio data and subtitle data and determining the code position of the test class object for the audio data and subtitle data, and generating a test API interface according to the code position;

[0166] Inputting the tooth movement sequence, the face movement sequence, the tongue movement sequence, the larynx movement sequence, the audio data, and the subtitle data into the test API interface for testing to obtain specific occupancy rate and power consumption change data of the intelligent terminal.

[0167] Preferably, the method further includes:

[0168] Inputting the tooth movement sequence, the face movement sequence, the tongue movement sequence, the larynx movement sequence, the audio data, and the subtitle data into an intelligent recognition model to obtain an output first matching prediction coefficient.

[0169] Preferably, identifying the tooth image, the face image, the tongue image, and the larynx image from the video data includes:

[0170] Decomposing the video data into multiple frame images;

[0171] Traverse the multiple frame images and divide the multiple frame images into portrait images and non-portrait images;

[0172] Identify the facial region of the portrait image;

[0173] Extract texture features for the facial region to obtain dental images, facial images, tongue images, and laryngeal images.

[0174] Preferably, inputting the dental motion sequence, facial motion sequence, tongue motion sequence, laryngeal motion sequence, audio data, and subtitle data into the test API interface for testing to obtain the specific occupancy rate and power consumption change data of the intelligent terminal, including:

[0175] Input the dental motion sequence, facial motion sequence, tongue motion sequence, laryngeal motion sequence, audio data, and subtitle data into the test API interface, and calculate to obtain a second matching prediction coefficient;

[0176] Adjust the second matching prediction coefficient according to the first matching prediction coefficient to obtain an adjusted matching prediction coefficient;

[0177] According to the adjusted matching prediction coefficient, correct the matching degree of the dental motion sequence, facial motion sequence, tongue motion sequence, laryngeal motion sequence, audio data, and subtitle data, and then perform testing to obtain the CPU occupancy rate, GPU occupancy rate, memory occupancy rate, and power consumption change data of the intelligent terminal.

[0178] Preferably, inputting the dental motion sequence, facial motion sequence, tongue motion sequence, laryngeal motion sequence, audio data, and subtitle data into the intelligent recognition model to obtain the output first matching prediction coefficient, including:

[0179] Convert the dental motion sequence, facial motion sequence, tongue motion sequence, laryngeal motion sequence, audio data, and subtitle data into a first feature vector through a feature extraction framework;

[0180] Perform downsampling and convolution operator transformation on the first feature vector to obtain a second feature vector;

[0181] Fuse the second feature vectors by means of channel dimension connection to obtain a fused feature;

[0182] Construct multiple neighbor nodes according to the fused feature to obtain a k-NN local graph;

[0183] Perform neighborhood aggregation operation on the edge features of the k-NN local graph to obtain an updated feature;

[0184] Input the updated feature into the binary classification module to obtain the trained intelligent recognition model. Input the new tooth movement sequence, facial movement sequence, tongue movement sequence, laryngeal movement sequence, audio data, and subtitle data into the trained intelligent recognition model, and output the first matching prediction coefficient.

[0185] Preferably, adjusting the second matching prediction coefficient according to the first matching prediction coefficient to obtain an adjusted matching prediction coefficient includes:

[0186] Convert and generate the adjustment amplitude of the audio track subtitle according to the first matching prediction coefficient;

[0187] Add the adjustment amplitude of the audio track subtitle to the second matching prediction coefficient to obtain the adjusted matching prediction coefficient.

[0188] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0189] Each embodiment in this specification is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other.

[0190] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the embodiments of the present invention can take the form of an all-hardware embodiment, an all-software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) that contain computer-usable program code.

[0191] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of systems, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing terminal devices generate a system for realizing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0192] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal devices to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including an instruction method that realizes the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0193] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal devices, such that a series of operation steps are executed on the computer or other programmable terminal devices to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable terminal devices provide steps for realizing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0194] Although the preferred embodiments of the embodiments of the present invention have been described, those skilled in the art can make additional changes and modifications to these embodiments once they learn the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0195] Finally, it should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, system, article or terminal device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, system, article or terminal device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, system, article or terminal device comprising the element.

[0196] The above has introduced in detail a method for audio and video testing of an intelligent terminal, an audio and video testing system of an intelligent terminal, an intelligent terminal and a storage medium provided by the present invention. Specific examples are used in this text to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A method for testing audio and video of an intelligent terminal, characterized in that: include: Acquire video data; wherein the video data includes video data and audio data; identifying a tooth image, a facial image, a tongue image, and a throat image from the video data; Extracting audio data and subtitle data corresponding to the teeth image, facial image, tongue image and throat image; Converting the tooth image, facial image, tongue image and laryngeal image into a tooth motion sequence, a facial motion sequence, a tongue motion sequence and a laryngeal motion sequence; Creating a test class object for the audio data and subtitle data and determining a code location of the test class object for the audio data and subtitle data, and generating a test API interface according to the code location; The tooth movement sequence, facial movement sequence, tongue movement sequence, throat movement sequence, audio data, and subtitle data are input into the test API interface for testing to obtain specific occupancy rate and power consumption change data of the smart terminal.

2. The method according to claim 1, characterized in that The method further comprises: The tooth movement sequence, facial movement sequence, tongue movement sequence, throat movement sequence, audio data, and subtitle data are input into an intelligent recognition model to obtain an output first matching prediction coefficient.

3. The method according to claim 1, characterized in that The step of identifying the teeth image, the face image, the tongue image and the throat image from the video data comprises: Decomposing the video data into a plurality of frame images; Traversing the plurality of frame images, and dividing the plurality of frame images into portrait images and non-portrait images; Recognizing the facial area of ​​the portrait image; Texture features are extracted for the facial region to obtain a tooth image, a facial image, a tongue image and a throat image.

4. The method according to claim 2, characterized in that: The tooth movement sequence, facial movement sequence, tongue movement sequence, throat movement sequence, audio data, and subtitle data are input into the test API interface for testing, and specific occupancy rate and power consumption change data of the smart terminal are obtained, including: Input the tooth movement sequence, facial movement sequence, tongue movement sequence, throat movement sequence, audio data, and subtitle data into the test API interface to calculate a second matching prediction coefficient; Adjusting the second matching prediction coefficient according to the first matching prediction coefficient to obtain an adjusted matching prediction coefficient; According to the adjustment of the matching prediction coefficient, the matching degree of the tooth movement sequence, facial movement sequence, tongue movement sequence and throat movement sequence and the audio data and subtitle data is corrected, and then tested to obtain the CPU occupancy rate, GPU occupancy rate, memory occupancy rate and power consumption change data of the smart terminal.

5. The method according to claim 2, characterized in that: The step of inputting the tooth movement sequence, facial movement sequence, tongue movement sequence, throat movement sequence, audio data, and subtitle data into the intelligent recognition model to obtain the output first matching prediction coefficient includes: The tooth movement sequence, the facial movement sequence, the tongue movement sequence, the throat movement sequence, the audio data, and the subtitle data are converted into a first feature vector through a feature extraction framework; Downsampling and convolution operator transforming the first eigenvector to obtain a second eigenvector; The second feature vectors are fused by connecting them in the channel dimension to obtain fused features; Construct multiple neighbor nodes based on the fusion features to obtain a k-NN local graph; Performing a neighborhood aggregation operation on edge features on the k-NN local graph to obtain updated features; The updated features are input into a binary classification module to obtain a trained intelligent recognition model, and new tooth movement sequences, facial movement sequences, tongue movement sequences, throat movement sequences, audio data, and subtitle data are input into the trained intelligent recognition model to output a first matching prediction coefficient.

6. The method according to claim 4, characterized in that The step of adjusting the second matching prediction coefficient according to the first matching prediction coefficient to obtain the adjusted matching prediction coefficient includes: Generate an audio track subtitle adjustment amplitude according to the first matching prediction coefficient conversion; The audio track subtitle adjustment amplitude is added to the second matching prediction coefficient to obtain the adjusted matching prediction coefficient.

7. An audio and video test system for an intelligent terminal, characterized in that: include: A video data acquisition module, used to acquire video data; wherein the video data includes video data and audio data; A recognition module, used to recognize teeth images, facial images, tongue images and throat images from the video data; An extraction module, used to extract audio data and subtitle data corresponding to the teeth image, facial image, tongue image and throat image; A conversion module, used for converting the tooth image, facial image, tongue image and laryngeal image into a tooth motion sequence, a facial motion sequence, a tongue motion sequence and a laryngeal motion sequence; A determination module, used to create a test class object of the audio data and subtitle data and determine the code location of the test class object of the audio data and subtitle data, and generate a test API interface according to the code location; The test module is used to input the tooth movement sequence, facial movement sequence, tongue movement sequence, throat movement sequence, audio data and subtitle data into the test API interface for testing, so as to obtain the specific occupancy rate and power consumption change data of the smart terminal.

8. The system according to claim 7, characterized in that The system further comprises: The output module is used to input the tooth movement sequence, facial movement sequence, tongue movement sequence, throat movement sequence, audio data and subtitle data into the intelligent recognition model to obtain the output first matching prediction coefficient.

9. An intelligent terminal, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the audio and video testing method for the smart terminal described in any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the audio and video testing method for a smart terminal according to any one of claims 1 to 6 are implemented.

Images