VINR user audio and video perception evaluation method

Abstract

The invention discloses a VINR user audio and video perception evaluation method, which comprises the following steps: S1, audio and video data initiation: a sending end initiates audio data and video data, and records and stores time sequence data corresponding to the audio data and the video data when the audio data and the video data are initiated; processing and storing the audio data, and storing the network data of the sending end at the same time; s2, receiving audio and video data: receiving the audio data and the video data by a receiving end, and processing and storing time sequences corresponding to the audio data and the video data when the audio data and the video data are recorded and received; processing and storing the audio data, and storing network data of a receiving end at the same time; s3, evaluation and analysis: evaluating various data of the sending end and the receiving end by using a similarity evaluation method, performing time sequence evaluation in combination with the time sequence data, and performing network quality evaluation in combination with the network data; and comprehensively obtaining audio and video perception comprehensive evaluation to form user perception evaluation. According to the method, the network problem is positioned more accurately.

Description

Technical Field

[0001] The present invention relates to the field of communications, and in particular to a video call service in the field of communications, and in particular to a method for evaluating audio and video perception of a ViNR user. Background Art

[0002] With the development of wireless communications, in the current 5G era, operators have launched the "5G New Call" service, which is an ultra-clear audio and video call service based on the 5G network, namely VoNR (Voice over New Radio) / ViNR (Video over New Radio). ViNR audio and video services will experience various distortions during the collection, compression, transmission and storage processes, and any distortion may lead to a decline in the quality of user voice and visual perception.

[0003] Currently, audio and video quality assessment methods in the industry are divided into two categories: voice quality assessment and image and video quality assessment.

[0004] Voice quality assessment is divided into two types: subjective assessment and objective assessment. ITU-TP.800 defines the subjective test method of MOS. The main objective test methods include PESQ, POLQA, etc. Among them, ITU-TP.863 (POLQA) is the method currently recommended by ITU for VoLTE voice quality testing.

[0005] Image and video evaluation can also be divided into two types: subjective evaluation and objective evaluation. Subjective quality evaluation mainly relies on human eyes to watch and score. It can be said that it is the most intuitive method to reflect the audience's perception of video quality and is also the ultimate goal of other objective evaluation methods. However, this method has problems such as time-consuming and labor-intensive, high cost, and subjective perception bias, and cannot be directly applied in the industry. Objective quality evaluation of image and video quality is usually modeled using quality assessment (QA) algorithms. Quality assessment algorithms can accurately measure the pros and cons of coding and decoding models, communication transmission systems, image enhancement and reconstruction algorithms. According to the recommendations of the International Telecommunication Union (ITU), they can be divided into five categories according to the input data type: media layer model, parametric packet layer model, parametric planning model, bitstream layer model, and hybrid model. Among them, the media layer model directly uses media information for calculation and analysis to give evaluation results, while other types of evaluation methods evaluate quality based on external variables such as coding parameters or network channel status.

[0006] Chinese patent document CN 107920362 A discloses a micro-area-based LTE network performance evaluation method, including the following steps: (1) Data collection: collecting user-level OTT information, MR data, key signaling switching data and traffic statistics data; (2) establishing a location fingerprint database; (3) Data processing: integrating and associating various data sources; at the same time, classifying the traffic statistics data in the two major service types of LTE and VoLTE according to five dimensions of retention, accessibility, integrity, cell integrity rate, and mobility, and marking the indicator attributes; (4) Data calculation and analysis; (5) Data analysis results: the service type is divided into two types of LTE (browsing service) and VoLTE service, and the time to be evaluated can be selected by the user. The grid network performance score is divided into five intervals of excellent, good, general, poor, and severe. By using the association and constraint relationship between the indicator sets in each dimension, the network quality of the micro-area can be reasonably and objectively evaluated, and network optimization can be effectively guided.

[0007] The existing quality assessment methods are methods for evaluating voice quality and image and video quality separately. The values ​​of individual indicators are either high or low, and cannot directly represent the user's subjective feelings, especially the audio and video quality feelings. Therefore, it is necessary to develop a ViNR user audio and video perception evaluation method to more accurately locate network problems. Summary of the invention

[0008] The technical problem to be solved by the present invention is to propose a ViNR user audio and video perception evaluation method, which not only solves the problem of poor repeatability of the MOS subjective evaluation method, but also solves the problem that the MOS-LQO objective problem cannot restore the human brain thinking paradigm. It is closer to the human brain thinking mode and closer to the user's perception of the quality of network audio and video calls. At the same time, through time and location mapping, combined with network parameters and events, network problems can be located more accurately.

[0009] In order to solve the above technical problems, the technical solution adopted by the present invention is: the ViNR user audio and video perception evaluation method specifically includes the following steps:

[0010] S1 audio and video data initiation: The sender initiates audio data and video data, and records and saves the time series data corresponding to the audio data and video data at the time of initiation; the audio data is processed and saved, and the network data of the sender is saved at the same time;

[0011] S2 audio and video data reception: The receiving end receives audio data and video data, processes and saves the time series corresponding to the audio data and video data recorded when receiving; the audio data is processed and saved, and the network data of the receiving end is saved at the same time;

[0012] S3 Evaluation and Analysis: Use similarity evaluation methods to evaluate various data from the sender and receiver, combine time series data for time series evaluation, and combine network data for network quality evaluation; finally, obtain a comprehensive evaluation of audio and video perception to form a user perception evaluation.

[0013] The above technical solution uses video similarity, audio similarity, text similarity, network quality evaluation and time series evaluation to conduct a comprehensive evaluation of audio and video perception, which not only solves the problem of poor repeatability of the MOS subjective evaluation method, but also solves the problem that the MOS-LQO objective problem cannot restore the human brain thinking paradigm. It is closer to the human brain thinking mode and the user's perception of the quality of network audio and video calls. At the same time, through time and location mapping, combined with network parameters and events, network problems can be more accurately located. The receiving end and the sending end are different clients or platforms. The method processes the audio and video data of the sending and receiving ends, converts the audio data of the sending and receiving ends into text information, and uses the video, audio and text data similarity fitting algorithm to perform similarity evaluation; and performs time series evaluation on the saved time series data of the sending and receiving ends through a time series alignment algorithm and method; displays and saves the network parameters and event information of the network connected to the communication unit of the sending and receiving end in real time, and performs network quality evaluation through a network quality evaluation algorithm and method; finally, combines video similarity, audio similarity, text similarity, network quality evaluation and time series evaluation to perform a comprehensive evaluation of audio and video perception to form a user perception evaluation; the method not only solves the problem of poor repeatability of subjective evaluation methods, but also solves the problem that objective problems cannot restore the human brain thinking paradigm, is closer to the human brain thinking mode, and is closer to the user's perception of the quality of network audio and video calls, and at the same time, through time and position mapping, combined with network parameter information and events, it can more accurately locate network problems.

[0014] Preferably, the specific steps of step S1 are:

[0015] S11: The sending end initiates video data, records time series data during the video data initiation, and uploads the recorded time series data to the sender-receiver time series storage module of the server through the communication network for storage;

[0016] S12: The sending end initiates audio data and processes the audio data, that is, converts the audio into text information of the sending end, and records the time series data and network data in the process; uploads the converted text information of the receiving end to the text information storage module of the server for storage; and uploads the recorded time series data to the time series storage module of the server for storage; uploads the recorded network data to the network data storage module of the server for storage;

[0017] S13: The video data and audio data sent by the sending end are stored in the storage module at the same time.

[0018] Preferably, the specific steps of step S2 are:

[0019] S21: The receiving end initiates video data, records time series data during the video data reception, and uploads the recorded time series data to the sender-receiver time series storage module of the server through the communication network for storage;

[0020] S22: The receiving end initiates audio data and processes the audio data, that is, converts the audio into receiving end text information, and records the time series data and network data in the process; uploads the converted receiving end text information to the text information storage module of the server for storage; and uploads the recorded time series data to the time series storage module of the server for storage; uploads the recorded network data to the network data storage module of the server for storage;

[0021] S23: The video data and audio data received by the receiving end are stored simultaneously through the storage module.

[0022] Preferably, the specific steps of step S3 are:

[0023] S31: using a video similarity method, evaluating the similarity between the video data of the sending end in step S1 and the video data of the receiving end in step S2, and displaying the results in real time;

[0024] S32: using a text similarity method, evaluating the similarity between the text information data converted from the audio data processed by the sending end in step S1 and the text information data converted from the audio data processed by the receiving end in step S2, and displaying them in real time;

[0025] S33: Use voice information to establish a user perception evaluation model through a telecommunication psychology algorithm to evaluate the user's voice perception;

[0026] S34: Evaluate network quality using a network quality evaluation algorithm and method based on network parameters and event information of the sending end and the receiving end;

[0027] S35: Combine steps S31, S32, S33 and S34 to perform a comprehensive evaluation of audio and video perception and finally form a user perception evaluation.

[0028] Preferably, the specific steps of evaluating the video similarity using the video similarity in step S31 are:

[0029] S311: The sending end divides the original video data according to the time stored in the time series data, so as to form an association relationship between each frame of the original image and a time point;

[0030] S312: The receiving end collects the comparison video of the original video data through the communication network, and performs frame segmentation on the comparison video according to the time of the video frames stored in the time series to form a correspondence between each frame of the comparison image and the comparison time point;

[0031] S313: Calculate the image similarity between each frame of the original image at each time point and each frame of the comparison image at the comparison time point, evaluate the video quality based on the image similarity calculation result, and finally output the result.

[0032] Preferably, the specific steps of evaluating the file similarity using text similarity in step S32 are:

[0033] S321: generating a corresponding standard audio as comparison audio by mechanically reading the original audio data of the sending end, and then converting the comparison audio into original text;

[0034] S322: The receiving end (another terminal or platform) collects the comparison audio through a communication network and converts it into a comparison text;

[0035] S323: Calculate the text similarity between the original text and the comparison text using a text similarity algorithm, then perform function mapping conversion, and finally output the result.

[0036] Preferably, the speech perception evaluation is performed by telecommunication psychology algorithm in step S33, that is, various speech samples are evaluated by artificial perception, and a user speech perception evaluation model is established. The specific steps of performing speech perception evaluation are:

[0037] S331 data collection: collects voice audio files of senders and receivers under different network qualities and corresponding VoLTE / VONR network indicators, including call establishment delay, jitter, voice packet loss rate, IP packet delay, and switching interruption delay;

[0038] S332 data processing: Users listen to the voice audio files of the voice initiator and the voice receiver respectively, and vote on the quality of the audio according to their personal perception. A corresponding threshold is set according to the voting result. If users who exceed the threshold give a good score, the audio file is labeled 1; label 0 means that users who exceed the threshold give a bad score. Therefore, each VoLTE / VONR network indicator has its corresponding perception label.

[0039] S333 Feature selection: Before building the classification model, the feature scores in xgboost are used to screen the final variables; feature variables are screened to prevent some variables from being too highly correlated;

[0040] S334 Model establishment: Use multiple classification algorithms to train the training set based on the network indicators corresponding to the existing good and bad audio, and use the test set for verification to obtain the optimal classification model, and output the user perception model;

[0041] S335 model prediction: perform user perception model prediction on the network indicators corresponding to the audio, and map the perception probability into a user perception score; in step S334, the score table for each audio file can be output through the established classification model.

[0042] Preferably, the specific steps of the network quality evaluation algorithm and method in step S34 are:

[0043] S341 Data Collection: Collect network data including network parameters and event information, i.e. user GPS information, MR data and VoLTE / VONR data;

[0044] S342 data processing: integrating and associating the data sources in step S322 at the grid level;

[0045] S343 Data calculation and analysis: Before calculating the grid network performance index items, first calculate the basic network performance scores of each cell covering the grid. After obtaining the basic network performance scores of all cells covering the grid, use the algorithm to obtain the basic network performance score of the grid.

[0046] S344 data analysis results: The service type is VoLTE / VONR service, and then select the time to be evaluated to obtain the grid network performance score. The grid network performance score is divided into five intervals: excellent, good, general, poor, and severe.

[0047] The threshold of each indicator is adjusted to truly reflect the current network quality, especially to realize the network performance evaluation of VoLTE / VONR on the 50*50 grid, which is more in line with the needs of mobile network optimization. The association and constraint relationship between indicator sets is used to enable reasonable and objective evaluation of the network quality of micro-areas (50*50 grids, hereinafter collectively referred to as grids), effectively guiding network optimization.

[0048] Preferably, the basic network performance score of the cell in step S343 The scores of all KPI indicators are: The weighted sum is obtained to calculate the basic network performance score of each cell in the coverage grid That is, the KPI score of each cell is calculated using different algorithms according to the indicator attributes, specifically:

[0049] If the indicator is smaller, the better, when When , the calculation formula is:

[0050]

[0051] in, KPI for all communities j The value of the 2.5%-97.5% quantile interval of the indicator, KPI for cell X j The range of the interval, where the numerator is the KPI in cell X j Cumulative distribution function (AUC), with KPI as the denominator j The value corresponding to the cell with the largest cumulative distribution function;

[0052] If the KPI of cell X j Less than When the left endpoint is , the calculation formula is:

[0053]

[0054] If the KPI of cell X j Greater than When the right endpoint is , the calculation formula is:

[0055]

[0056] If the indicator is bigger, the better, when When , the calculation formula is:

[0057]

[0058] If the KPI of cell X j Greater than The right endpoint of , the calculation formula is:

[0059]

[0060] If the KPI of cell X j Less than The left endpoint of , the calculation formula is:

[0061]

[0062] Finally, the basic network performance score of all cells in the coverage grid is obtained;

[0063] In step S343, the basic network performance scores of all cells covering the grid are obtained through data calculation and analysis, and then the basic network performance scores of the grid are obtained by using an algorithm. The specific algorithm is:

[0064]

[0065] Among them, Grid X refers to a grid X, Refers to the set of all cells covering grid X;

[0066] After obtaining the basic performance score of the grid based on the above algorithm logic, the MR data based on the user in the grid is added as the adjustment parameter Thus, the final basic network performance score of each grid is obtained, and the formula is:

[0067]

[0068] The range of this adjustment parameter is in is the value of grid X corresponding to the normalized value of the RSRP mean of all grids for 14 consecutive days, For each grid, the SINR mean value for a certain 14-day period is normalized to the min-max value corresponding to the SINR mean value of the grid;

[0069] Normalization of min-max is also called deviation normalization, which is a linear transformation of the original data so that the result is mapped between 0 and 1. The conversion function is:

[0070]

[0071] Among them, max is the maximum value of the sample data, and min is the minimum value of the sample data;

[0072] Finally, the final basic network performance score of the grid is obtained based on the basic network performance score of the grid and the adjustment parameters. The formula is:

[0073]

[0074] Finally, the basic network performance score The score is mapped to the interval (0, 100).

[0075] Preferably, the method for performing comprehensive evaluation of audio and video perception in step S35 specifically includes the following steps:

[0076] After obtaining three user speech perception scores through video perception evaluation, speech perception evaluation, network quality evaluation and text similarity respectively, different weights are set for the results obtained by the three methods according to experience, and the final user speech perception score is obtained by weighted average; the weight based on the video perception evaluation method is SW0, the weight based on the speech perception evaluation method is SW1, the weight based on the network quality evaluation method is SW2, and the weight based on the text similarity method is SW3; the final user speech perception comprehensive evaluation; the calculation formula is:

[0077] S ensemble =SW0*S0+SW1*S1+SW2*S2+SW3*S3;

[0078] Among them: S0 is the scoring result based on the video perception evaluation method, S1 is the scoring result based on the speech perception evaluation method, S2 is the scoring result based on the network quality evaluation method, and S3 is the scoring result based on the text similarity method.

[0079] Preferably, four text similarity algorithms are used in step S323, and the text similarity algorithms in step S33 include four text similarity algorithms, namely: a statistical algorithm based on word frequency (TF), a Simhash text similarity algorithm, a text similarity algorithm based on a vector space model VSM, and a text similarity algorithm based on an LDA topic model;

[0080] The specific steps of the statistical algorithm based on word frequency (TF) are as follows:

[0081] S332-1-1: List each word of the standard text;

[0082] S332-1-2: Count the frequency of each word in the standard text in the standard text and the comparison text respectively. s 、f c , where f c ≤f s , the redundant words in the comparison text are not counted in the statistics;

[0083] S332-1-3: Text similarity result, the formula is: Similarity tf is the similarity of word frequency TF;

[0084] The specific steps of the Simhash text similarity algorithm include:

[0085] S332-2-1: Segment the text into words respectively, and take the features and weights of the first n words with the highest TF-IDF weights in the text; that is, a set of (feature: weight) with a length of n is obtained for each text;

[0086] S332-2-2: Perform a common hash on the feature of the word to obtain a 64-bit binary, and obtain a set of (hash: weight) with a length of 20;

[0087] S332-2-3: According to whether the corresponding position in the binary hash obtained in step S332-2-2 is 1 or 0, a positive weight or a negative weight is taken for the corresponding position; for example, a word is obtained by step S332-2-2 (010111:5), and after step S332-2-3, a list [-5,5,-5,5,5,5] can be obtained, that is, 20 lists of length 64 [weight,-weight…weight] are obtained for a document;

[0088] S332-2-4: Perform column vector addition on the n lists in step S332-2-3 to obtain a list; for example, [-5,5,-5,5,5,5], [-3,-3,-3,3,-3,3], [1,-1,-1,1,1,1] are added by column vector to obtain [-7,1,-9,9,3,9], thus obtaining a list of length 64 for one document;

[0089] S332-2-5: judge each value in the list obtained in step S332-2-4, and take 0 when it is a negative value, and take 1 when it is a positive value; for example, [-7,1,-9,9,3,9] gets 010111, so a list of length 64 is obtained for a text;

[0090] S332-2-6: Calculate similarity. Take the XOR of the simhash of the two texts. If it is 1, it means they are different, and 0 means they are the same. The length of 1 divided by the total length is the difference, and 1 minus the difference is the text similarity. simhash ;

[0091] The basic idea of ​​VSM is to simplify the text into an N-dimensional vector representation with the weight of the feature item (keyword) as the component. The model assumes that words are unrelated and uses vectors to represent the text, thereby simplifying the complex relationship between keywords in the text. The text is represented by a very simple vector, making the model computable. D is used as the abbreviation of Document to represent the text, and T is used as the abbreviation of Term to represent the feature item; the feature item refers to the basic language unit that appears in the document D and can represent the content of the document, mainly composed of words or phrases. The text can be represented by the feature item set D (T1, T2, ..., T n ), where T k is a feature item, which is required to satisfy 1<=k<=N; the specific steps of the text similarity algorithm based on the vector space model VSM include:

[0092] S332-3-1: Assume that a speech text has four feature items a, b, c, and d, then the speech text is represented by D(a, b, c, d);

[0093] S332-3-2: For other texts to be compared with the speech text, the order of the feature items is also followed; for a text containing n feature items, a certain weight is given to each feature item to indicate its importance, that is, D = D (T1, W1; T2, W2; ..., T n , W n ) is abbreviated as D = D (W1, W2, ..., W n ), is called the weight vector representation of the text D; where W k It is T k The weight of , 1<=k<=N;

[0094] S332-3-3: In the vector space model, the content relevance Sim(D1, D2) between text D1 and text D2 is expressed by the cosine value of the angle between the vectors, and the formula is:

[0095]

[0096] The specific steps of the text similarity algorithm based on the LDA topic model include: firstly, using the LDA model to model the text set, that is, using the statistical characteristics of the text, mapping the text corpus to each topic space, mining the relationship between different topics and words hidden in the text, obtaining the topic distribution of the text, and then calculating the similarity matrix of the text through the topic distribution; the LDA model is a probabilistic topic model for modeling discrete data sets (such as document sets), and is a method for modeling the topic information of text data. By making a brief description of the text and retaining the essential statistical information, it helps to efficiently process large-scale document sets;

[0097] The process of generating text by the LDA topic probability model is as follows:

[0098] S332-4-1: For topic z, a word multinomial distribution vector on the topic is obtained according to the Dirichlet distribution Dir(β)

[0099] S332-4-2: Obtain the number of words N in the text according to Poisson distribution P;

[0100] S332-4-3: Obtain a topic distribution probability vector θ of the text according to Dirichlet distribution Dir(α);

[0101] S332-4-4: For each word Wn in the N words of the text:

[0102] S332-4-5: Randomly select a topic z from the multinomial distribution Multinomial(θ) of θ;

[0103] S332-4-6: Select a word as Wn from the multinomial conditional probability distribution Multinomial(Φ) of topic z;

[0104] Since the topic distribution of a text is a simple mapping of the text vector space, in the case of the topic representation of the text, the similarity between two texts is calculated by calculating the corresponding topic probability distribution; since the topic is a mixed distribution of word vectors, the Kullback–Leibler relative entropy distance, denoted as KL distance, is used as the similarity metric. The calculation formula of KL distance is: Where D KL (p, q) represents the information loss when the probability distribution Q is used to fit the true distribution P, where P represents the true distribution and q represents the fitted distribution of P.

[0105] Preferably, the system based on ViNR user audio and video perception evaluation includes a sending end module, a receiving end module and a perception evaluation module, the sending end module is connected to the receiving end module through a communication network, and the sending end module and the receiving end module are respectively communicated with the perception evaluation module; the sending end module includes a sending end audio and video data unit and a communication unit 1, and the communication unit 1 forms a data connection with the sending end audio and video data unit and the sending end audio and video data processing unit respectively; the receiving end module includes a receiving end audio and video data unit and a communication unit 2, and the communication unit 2 forms a data connection with the receiving end audio and video data unit and the receiving end audio and video data processing unit respectively; the perception evaluation module includes a sending end audio and video data processing unit, a data time series processing unit, a receiving end audio and video data processing unit, a video storage unit, a video similarity evaluation unit, an audio storage unit, an audio similarity evaluation unit, a time series storage unit The transmitting end module is connected to the transmitting end audio and video data processing unit of the perception evaluation module through a communication unit 1, and the receiving end audio and video data processing unit is connected to the receiving end audio and video data processing unit through a communication unit 2. The transmitting end module is connected to the transmitting end audio and video data processing unit of the perception evaluation module through a communication unit 1, and the receiving end module is connected to the receiving end audio and video data processing unit of the perception evaluation module through a communication unit 2.

[0106] The above technical scheme is adopted, wherein the video storage unit is used to store the video information processed by the sending end audio and video data processing unit and the receiving end audio and video data processing unit, the audio storage unit is used to store the audio information processed by the sending end audio and video data processing unit and the receiving end audio and video data processing unit, the time series storage unit is used to store the time series information processed by the sending end audio and video data processing unit and the receiving end audio and video data processing unit, the text storage unit is used to store the text information processed by the sending end audio and video data processing unit and the receiving end audio and video data processing unit, and the signal storage module is used to store the network parameters and event information of the sending end module and the receiving end module; the audio and video sending end module + the audio and video receiving end module + the perception evaluation module are combined into a network user audio and video perception evaluation system, thereby realizing the audio and video perception evaluation of network users.

[0107] Compared with the prior art, the present invention has the following beneficial effects: using video similarity, audio similarity, and text similarity algorithms to judge the user-perceived audio and video quality, it not only solves the problem of poor repeatability of the MOS subjective evaluation method, but also solves the problem that the MOS-LQO objective problem cannot restore the human brain thinking paradigm. It is closer to the human brain thinking mode and closer to the user's perception of the audio and video quality of network calls. At the same time, through time and location mapping, combined with network parameters and events, network problems can be located more accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0108] Figure 1 It is a schematic diagram of a method and flow chart of evaluating the audio and video perception of a ViNR user of the present invention;

[0109] Figure 2 It is a flow chart of video similarity evaluation of the ViNR user audio and video perception evaluation method of the present invention;

[0110] Figure 3 It is a flowchart of text similarity evaluation of the ViNR user audio and video perception evaluation method of the present invention;

[0111] Figure 4 It is a system framework diagram of the ViNR user audio and video perception evaluation method of the present invention. DETAILED DESCRIPTION

[0112] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiment diagram of the present invention.

[0113] Example: Figure 1 As shown, the ViNR user audio and video perception evaluation method specifically includes the following steps:

[0114] S1 audio and video data initiation: The sender initiates audio data and video data, and records and saves the time series data corresponding to the audio data and video data at the time of initiation; the audio data is processed and saved, and the network data of the sender is saved at the same time;

[0115] The specific steps of step S1 are:

[0116] S11: The sending end initiates video data, records time series data during the video data initiation, and uploads the recorded time series data to the sender-receiver time series storage module of the server through the communication network for storage;

[0117] S12: The sending end initiates audio data, processes the audio data, and converts the audio into text information of the sending end, and records the time series data and network data of the process; uploads the converted text information of the receiving end to the text information storage module of the server through the communication network for storage; and uploads the recorded time series data to the time series storage module of the server through the communication network for storage; uploads the recorded network data to the network data storage module of the server through the communication network for storage;

[0118] S13: The video data and audio data sent by the sending end are stored in the storage module at the same time;

[0119] S2 audio and video data reception: The receiving end receives audio data and video data, processes and saves the time series corresponding to the audio data and video data recorded when receiving; the audio data is processed and saved, and the network data of the receiving end is saved at the same time;

[0120] The specific steps of step S2 are:

[0121] S21: The receiving end initiates video data, records time series data during the video data reception, and uploads the recorded time series data to the sender-receiver time series storage module of the server through the communication network for storage;

[0122] S22: The receiving end initiates audio data, processes the audio data, and converts the audio into receiving end text information, while recording the time series data and network data in the process; uploads the converted receiving end text information to the text information storage module of the server through the communication network for storage; and uploads the recorded time series data to the time series storage module of the server through the communication network for storage; uploads the recorded network data to the network data storage module of the server through the communication network for storage;

[0123] S23: simultaneously storing the received video data and audio data at the receiving end through the storage module;

[0124] S3 Evaluation and Analysis: Use similarity evaluation methods to evaluate various data from the sender and receiver, combine time series data for time series evaluation, and combine network data for network quality evaluation; finally, obtain a comprehensive audio and video perception evaluation to form a user perception evaluation;

[0125] The specific steps of step S3 are:

[0126] S31: using a video similarity method, evaluating the similarity between the video data of the sending end in step S1 and the video data of the receiving end in step S2, and displaying the results in real time;

[0127] like Figure 2 As shown, the specific steps of using video similarity to evaluate video similarity in step S31 are:

[0128] S311: The sending end divides the original video data according to the time stored in the time series data, so as to form an association relationship between each frame of the original image and a time point;

[0129] S312: The receiving end (i.e., another terminal or platform different from the sending end) collects the comparison video of the original video data through the communication network, and performs frame segmentation on the comparison video according to the time of the video frames stored in the time series to form a correspondence between each frame of the comparison image and the comparison time point;

[0130] S313: Calculate the image similarity between each frame of the original image at each time point and each frame of the comparison image at the comparison time point, evaluate the video quality based on the image similarity calculation result, and finally output the result; in order to more accurately evaluate the video quality and be close to the actual perception of the user, we selected the image similarity method to evaluate the video quality; Calculate the image similarity between the comparison video and the original video. The present invention relates to a set of algorithm frameworks for calculating the similarity of images. The specific steps of step S313 are:

[0131] S3131: Preprocess the image, scale the image, and adjust the image to a uniform size to facilitate subsequent feature extraction and comparison;

[0132] S3132: grayscale processing is performed to convert the color image into a grayscale image, thereby reducing the influence of color on similarity calculation and reducing calculation complexity;

[0133] S3133: normalize the image to eliminate the influence of factors such as illumination on the image similarity;

[0134] S3134: SIFT extracts key points or feature vectors from an image. These features should be able to reflect the main information of the image, such as shape, texture, color, etc. According to the specific algorithm, the descriptor or feature vector of each key point is calculated;

[0135] In S3134, the algorithm SIFT (Scale-Invariant Feature Transform) for extracting key points from an image is mentioned. The algorithm is a feature extraction algorithm used in image processing and computer vision. The specific steps of using the SIFT algorithm are:

[0136] S31341 Scale Space Pyramid: Filter the original image using Gaussian blur functions of different scales to generate a series of image pyramids, also known as scale space;

[0137] S31342 Keypoint Localization: Use Difference of Gaussian (DoG) scale space to detect potential keypoint locations; keypoints are stable in space and scale, and they can maintain a certain invariance under different scales and rotations;

[0138] S31343 Key point direction determination: After determining the potential key point locations, the SIFT algorithm calculates the gradient magnitude and direction of the area around each key point and uses the gradient histogram to determine the main direction of the key point;

[0139] S31344 Key point descriptor generation: Generate a descriptor vector for each key point based on the position, scale and orientation information of the key point, which contains detailed information about the image area around the key point for subsequent feature matching;

[0140] S31345 Feature Matching: Use the generated descriptors to search for feature points in the target image that are similar to the key points in the reference image;

[0141] S3135 Find key points: compare the features of the image and the original image to find their similar key points;

[0142] S3136 calculates similarity: calculates similarity scores based on matched feature points, such as based on distance measurement (Euclidean distance) or based on similarity measurement (cosine similarity, etc.); the step S336 mentions based on similarity measurement (cosine similarity), which is a method commonly used to calculate the similarity between two vectors, and its basis is the cosine value of the angle between the vectors; if the angle between the two vectors is smaller, then their cosine values ​​are larger, indicating that the two vectors are more similar; specifically:

[0143] S31361: First, the descriptor vectors of the original image and the contrast image are obtained and set as vector A and vector B respectively;

[0144] S31362: Calculate the cosine similarity. The calculation formula of the cosine similarity is:

[0145]

[0146] S31363: Calculate the dot product by multiplying the elements at corresponding positions of vector A and vector B, and then adding all the products;

[0147] S31364: Add the squares of each element in the vector and then take the square root. Use the above formula to calculate the cosine similarity;

[0148] S3137: Calculate the similarity of the two images based on the result of feature matching. For the missing comparison image, set the similarity to 0;

[0149] S3138: Accumulate the similarity of each frame and calculate the average value as the final video similarity result;

[0150] S32: using a text similarity method, evaluating the similarity between the text information data converted from the audio data processed by the sending end in step S1 and the text information data converted from the audio data processed by the receiving end in step S2, and displaying them in real time;

[0151] like Figure 3 As shown, the specific steps of using text similarity to evaluate file similarity in step S32 are:

[0152] S321: generating a corresponding standard audio as comparison audio by mechanically reading the original audio data of the sending end, and then converting the comparison audio into original text;

[0153] S322: The receiving end (another terminal or platform) collects the comparison audio through a communication network and converts it into a comparison text;

[0154] S323: Calculate the text similarity between the original text and the comparison text using a text similarity algorithm, then perform function mapping conversion, and finally output the result;

[0155] Four text similarity algorithms are used in step S323, and the text similarity algorithms in step S33 include four text similarity algorithms, namely: a statistical algorithm based on word frequency (TF), a Simhash text similarity algorithm, a text similarity algorithm based on a vector space model VSM, and a text similarity algorithm based on an LDA topic model;

[0156] The specific steps of the statistical algorithm based on word frequency (TF) are as follows:

[0157] S332-1-1: List each word of the standard text;

[0158] S332-1-2: Count the frequency of each word in the standard text in the standard text and the comparison text respectively. s 、f c , where f c ≤f s , the redundant words in the comparison text are not counted in the statistics;

[0159] S332-1-3: Text similarity result, the formula is: Similarity tf is the similarity of word frequency TF;

[0160] The specific steps of the Simhash text similarity algorithm include:

[0161] S332-2-1: Segment the text into words respectively, and take the features and weights of the first n words with the highest TF-IDF weights in the text; that is, a set of (feature: weight) with a length of n is obtained for each text;

[0162] S332-2-2: Perform a common hash on the feature of the word to obtain a 64-bit binary, and obtain a set of (hash: weight) with a length of 20;

[0163] S332-2-3: According to whether the corresponding position in the binary hash obtained in step S332-2-2 is 1 or 0, a positive weight or a negative weight is taken for the corresponding position; for example, a word is obtained by step S332-2-2 (010111:5), and after step S332-2-3, a list [-5,5,-5,5,5,5] can be obtained, that is, 20 lists of length 64 [weight,-weight…weight] are obtained for a document;

[0164] S332-2-4: Perform column vector addition on the n lists in step S332-2-3 to obtain a list; for example, [-5,5,-5,5,5,5], [-3,-3,-3,3,-3,3], [1,-1,-1,1,1,1] are added by column vector to obtain [-7,1,-9,9,3,9], thus obtaining a list of length 64 for one document;

[0165] S332-2-5: judge each value in the list obtained in step S332-2-4, and take 0 when it is a negative value, and take 1 when it is a positive value; for example, [-7,1,-9,9,3,9] gets 010111, so a list of length 64 is obtained for a text;

[0166] S332-2-6: Calculate similarity. Take the XOR of the simhash of the two texts. If it is 1, it means they are different, and 0 means they are the same. The length of 1 divided by the total length is the difference, and 1 minus the difference is the text similarity. simhash ;

[0167] The basic idea of ​​VSM is to simplify the text into an N-dimensional vector representation with the weight of the feature item (keyword) as the component; the model assumes that words are unrelated and uses vectors to represent the text, thereby simplifying the complex relationship between keywords in the text. The text is represented by a very simple vector, making the model computable; D is used as the abbreviation of Document to represent the text, and T is used as the abbreviation of Term to represent the feature item; the feature item refers to the basic language unit that appears in the document D and can represent the content of the document, which is mainly composed of words or phrases. The text can be represented by the feature item set D(T1, T2, ..., T n ), where T k is a feature item, which is required to satisfy 1<=k<=N; the specific steps of the text similarity algorithm based on the vector space model VSM include:

[0168] S332-3-1: Assume that a speech text has four feature items a, b, c, and d, then the speech text is represented by D(a, b, c, d);

[0169] S332-3-2: For other texts to be compared with the speech text, the order of the feature items is also followed; for a text containing n feature items, a certain weight is given to each feature item to indicate its importance, that is, D = D (T1, W1; T2, W2; ..., T n , W n ) is abbreviated as D = D (W1, W2, ..., W n ), is called the weight vector representation of the text D; where W k It is T k The weight of , 1<=k<=N;

[0170] S332-3-3: In the vector space model, the content relevance Sim(D1, D2) between text D1 and text D2 is expressed by the cosine value of the angle between the vectors, and the formula is:

[0171]

[0172] The specific steps of the text similarity algorithm based on the LDA topic model include: firstly, using the LDA model to model the text set, that is, using the statistical characteristics of the text, mapping the text corpus to each topic space, mining the relationship between different topics and words hidden in the text, obtaining the topic distribution of the text, and then calculating the similarity matrix of the text through the topic distribution; the LDA model is a probabilistic topic model for modeling discrete data sets (such as document sets), and is a method for modeling the topic information of text data. By making a brief description of the text and retaining the essential statistical information, it helps to efficiently process large-scale document sets;

[0173] The process of generating text by the LDA topic probability model is as follows:

[0174] S332-4-1: For topic z, a word multinomial distribution vector on the topic is obtained according to the Dirichlet distribution Dir(β)

[0175] S332-4-2: Obtain the number of words N in the text according to Poisson distribution P;

[0176] S332-4-3: Obtain a topic distribution probability vector θ of the text according to Dirichlet distribution Dir(α);

[0177] S332-4-4: For each word Wn in the N words of the text:

[0178] S332-4-5: Randomly select a topic z from the multinomial distribution Multinomial(θ) of θ;

[0179] S332-4-6: Select a word as Wn from the multinomial conditional probability distribution Multinomial(Φ) of topic z;

[0180] Since the topic distribution of a text is a simple mapping of the text vector space, in the case of the topic representation of the text, the similarity between two texts is calculated by calculating the corresponding topic probability distribution; since the topic is a mixed distribution of word vectors, the Kullback–Leibler relative entropy distance, denoted as KL distance, is used as the similarity metric. The calculation formula of KL distance is: Where D KL (p, q) represents the information loss when the probability distribution Q is used to fit the true distribution P, where P represents the true distribution and q represents the fitted distribution of P;

[0181] S33: Use voice information to establish a user perception evaluation model through a telecommunication psychology algorithm to evaluate the user's voice perception;

[0182] In step S33, the speech perception evaluation is performed by the telecommunication psychology algorithm, and various speech samples are evaluated by artificial perception to establish a user speech perception evaluation model. The specific steps of the speech perception evaluation are as follows:

[0183] S331 data collection: collects voice audio files of senders and receivers under different network qualities and corresponding VoLTE / VONR network indicators, including call establishment delay, jitter, voice packet loss rate, IP packet delay, and switching interruption delay;

[0184] S332 data processing: Users listen to the voice audio files of the voice initiator and the voice receiver respectively, and vote on the quality of the audio according to their personal perception. A corresponding threshold is set according to the voting result. If users who exceed the threshold give a good score, the audio file is labeled 1; label 0 means that users who exceed the threshold give a bad score. Therefore, each VoLTE / VONR network indicator has its corresponding perception label.

[0185] S333 Feature selection: Before building the classification model, the feature scores in xgboost are used to screen the final variables; feature variables are screened to prevent some variables from being too highly correlated;

[0186] S334 Model establishment: Use multiple classification algorithms to train the training set based on the network indicators corresponding to the existing good and bad audio, and use the test set for verification to obtain the optimal classification model, and output the user perception model;

[0187] The multiple classification algorithms in step S334 include 4 classification algorithms: 1) decision tree; 2) random forest; 3) logistic regression; 4) XGBoost algorithm; wherein,

[0188] 1) The specific steps of the decision tree algorithm are:

[0189] S334-1-1: Select an optimal predictor variable to divide all sample units into two categories to maximize the purity of the two categories; if the predictor variable is continuous, select a split point for classification to maximize the purity of the two categories; if the predictor variable is a categorical variable, merge and reclassify the categories;

[0190] S334-1-2: Continue to execute the steps in step S334-1-1 for each subcategory;

[0191] S334-1-3: Repeat steps S334-1-1 to S334-1-2 until the number of sample units in the subcategory is too small or no classification method can reduce the impurity below a given threshold; the subcategories in the final set are the terminal nodes; determine the category to which each terminal node belongs based on the mode of the number of sample units in the terminal node.

[0192] S334-1-4: For any sample unit, execute the decision tree to obtain its terminal node, that is, the category to which the model prediction belongs can be obtained according to step S334-1-3; however, usually a very large tree will be obtained by this algorithm, resulting in overfitting and poor classification performance for units outside the training set; to solve the above problem, the 10-fold cross-validation method can be used to select the tree with the smallest prediction error.

[0193] 2) Random forest: A random forest is an ensemble classifier composed of a set of decision tree classifiers {h(X, θ k ), k = 1, 2,..., K}, where {θ k} is a random vector that follows an independent and identical distribution, and K represents the number of decision trees in the random forest. Given the independent variable X, each decision tree classifier determines the optimal classification result through voting; random forest involves sampling the sample units and variables to generate a large number of decision trees; for each sample unit, all decision trees classify it in turn; the specific steps of the random forest algorithm are as follows:

[0194] S334-2-1: Apply the bootstrap method to randomly draw K new bootstrap sample sets from the training set with replacement, and construct K classification trees from them. Each time, the samples not drawn form K out-of-bag data.

[0195] S334-2-2: Randomly draw m < M variables at each node of each tree. By calculating the information content contained in each variable, then select the variable with the most classification ability among the m variables for node splitting.

[0196] S334-2-3: Completely generate all decision trees without pruning.

[0197] S334-2-4: The category to which the terminal node belongs is determined by the mode category corresponding to the node.

[0198] S334-2-5: For a new observation point, classify it with all the trees, and its category is generated by the majority decision principle.

[0199] 3) The specific steps of the logistic regression algorithm are as follows:

[0200] S334-3-1 Establish prediction function: First, construct a suitable prediction function, expressed as h function. The function h is the classification function to be found. The output of the function h must be two values, which are used to predict the judgment result of the input data. Therefore, the Logistic function is used. The function form is:

[0201]

[0202] Next, we need to determine the boundary type for data partitioning. Here we only discuss the case of linear boundaries. For linear boundaries, the form is as follows:

[0203]

[0204] Where θ represents the regression parameter and x represents the independent variable;

[0205] The prediction function is constructed as:

[0206]

[0207] Where θ represents the regression parameter and x represents the independent variable;

[0208] h θ The value of the (x) function represents the probability of the result being 1, so the probability of the classification result for input x being class 1 and class 0 is calculated according to the following formula:

[0209] p(y|x;θ)=(h θ (x)) y (1-h θ (x)) 1-y y=1,0;

[0210] S33432 Establish Cost function: any value h that can measure the model prediction θ The difference function between (x) and the true value y is called the cost function; for each algorithm, the cost function is not unique, and the following selects the common cross entropy; after determining the function, by constantly changing the parameter θ, a smaller cost function value J(θ) is obtained;

[0211]

[0212] Among them, m is the number of training samples, h θ (x) is the predicted value, y is the true value;

[0213] 4) The specific steps of the XGBoost algorithm are:

[0214] S334-4-1 defines the complexity of a tree: First, split the tree into a structure part q and a leaf node weight part w, where w is a vector representing the output value of each leaf node, and T represents the number of leaf nodes in a decision tree;

[0215] f t (x) = w q(x) , w∈R T ,q:R d →{1, 2, ..., T};

[0216] Introducing the regularization term Ω(f t ) to control the complexity of the tree, thereby effectively controlling the overfitting of the model;

[0217]

[0218] Among them, T represents the number of leaf nodes of a decision tree, γ represents the coefficient that controls the complexity of the tree, which is equivalent to pre-pruning the tree of the XGBoost algorithm model, and λ represents the proportion by which the regularization term is changed, which is equivalent to giving a penalty to the complex model to prevent the model from overfitting;

[0219] S334-4-2 Boosting Tree model in XGBoost: Like the GBDT method, the XGBoost boosting model also uses residuals. The difference is that the split node selection is not necessarily the least square loss. Its loss function is as follows. Compared with GBDT, it adds a regularization term according to the complexity of the tree model:

[0220]

[0221] in, represents the estimated value, y i represents the true value, represents the model residual, Ω(f k ) is the regularization term mentioned earlier;

[0222] S334-4-3 Rewrite the objective function: In XGB oos In t, Taylor expansion is used to expand the loss function into a binomial function, provided that the loss function is first-order and second-order; continuous and differentiable. Assuming that our leaf node area is:

[0223] I j ={i|q(x i ) = j};

[0224] Among them, I j ={i|q(x i)=j} represents the label set of the samples assigned to the jth leaf node in the training sample. For example, the 1st, 3rd, and 5th samples in the training sample are assigned to the 2nd leaf node, then j={1, 3, 5};

[0225] For g i and h i They are defined as follows:

[0226]

[0227] Among them, y i represents the true value, represents the predicted value of the t-1 iteration;

[0228] Then the objective function of t trees can be converted into:

[0229]

[0230] definition

[0231] At this time, w j Take the derivative and set it to 0; we get:

[0232]

[0233]

[0234] S334-4-4 Scoring function of tree structure: The Obj value above represents the maximum reduction on the target when specifying a tree structure, which can be called the structural score; it can be considered as a more general function for scoring tree structures similar to the Gini index; for the tree structure with the smallest Obj score, the greedy method is used. Each time, an attempt is made to split the existing leaf nodes, the first leaf node is the root node, and the gain after the split is obtained:

[0235]

[0236] The formula can be decomposed into the score on the left leaf, the score on the right leaf, the score on the original leaf, and the regularization on the additional leaf; here, Gain is used as the condition for determining whether to split;

[0237] If Gain < 0, this leaf node will not be split. However, all the splitting schemes still need to be listed for each split. In practice, all samples g are first i Sort from small to large, and then traverse to see if each node needs to be split. This way of splitting only requires scanning the sample once to split GL and GR, and then split according to the Gain score;

[0238] S335 model prediction: predicting the user perception model for the network indicators corresponding to the audio, and mapping the perception probability to the user perception score; the score table of each audio file can be output through the classification model established in step S334; that is, the score table of each audio file can be output through the classification model established in step S334;

[0239] S34: Perform network quality evaluation based on the network parameters and event information of the sending end and the receiving end through a network quality evaluation algorithm and method; in order to evaluate the user network quality through the network parameters and event information storage of the voice initiator and the voice receiving end, the specific steps of the network quality evaluation algorithm and method in step S34 are as follows:

[0240] S341 Data Collection: Collect network data including network parameters and event information, i.e. user GPS information, MR data and VoLTE / VONR data;

[0241] S342 data processing: integrating and associating the data sources in step S322 at the grid level;

[0242] S343 Data calculation and analysis: Before calculating the grid network performance index items, first calculate the basic network performance scores of each cell covering the grid. After obtaining the basic network performance scores of all cells covering the grid, use the algorithm to obtain the basic network performance score of the grid.

[0243] The basic network performance score of the cell in step S343 The scores of all KPI indicators are: The weighted sum is obtained to calculate the basic network performance score of each cell in the coverage grid That is, the KPI score of each cell is calculated using different algorithms according to the indicator attributes, specifically:

[0244] If the indicator is smaller, the better. When , the calculation formula is:

[0245]

[0246] in, KPI for all communities j The value of the 2.5%-97.5% quantile interval of the indicator, KPI for cell X j The range of the interval, where the numerator is the KPI in cell X j Cumulative distribution function (AUC), with KPI as the denominator j The value corresponding to the cell with the largest cumulative distribution function;

[0247] If the KPIj of cell X is less than When the left endpoint is , the calculation formula is:

[0248]

[0249] If the KPI of cell X j Greater than When the right endpoint is , the calculation formula is:

[0250]

[0251] If the indicator is bigger, the better, when When , the calculation formula is:

[0252]

[0253] If the KPI of cell X j Greater than The right endpoint of , the calculation formula is:

[0254]

[0255] If the KPI of cell X j Less than The left endpoint of , the calculation formula is:

[0256]

[0257] Finally, the basic network performance score of all cells in the coverage grid is obtained;

[0258] In step S343, the basic network performance scores of all cells covering the grid are obtained through data calculation and analysis, and then the basic network performance scores of the grid are obtained by using an algorithm. The specific algorithm is:

[0259]

[0260] Among them, Grid X refers to a grid X, Refers to the set of all cells covering grid X;

[0261] After obtaining the basic performance score of the grid based on the above algorithm logic, the MR data based on the user in the grid is added as the adjustment parameter Thus, the final basic network performance score of each grid is obtained, and the formula is:

[0262]

[0263] The range of this adjustment parameter is in is the value of grid X corresponding to the normalized value of the RSRP mean of all grids for 14 consecutive days, For each grid, the SINR mean value for a certain 14-day period is normalized to the min-max value corresponding to the SINR mean value of the grid;

[0264] Normalization of min-max is also called deviation normalization, which is a linear transformation of the original data so that the result is mapped between 0 and 1. The conversion function is:

[0265]

[0266] Among them, max is the maximum value of the sample data, and min is the minimum value of the sample data;

[0267] Finally, the final basic network performance score of the grid is obtained based on the basic network performance score of the grid and the adjustment parameters. The formula is:

[0268]

[0269] Finally, the basic network performance score The score is mapped to the interval of (0, 100);

[0270] S344 data analysis results: The service type is VoLTE / VONR service, and then the time to be evaluated is selected to obtain the grid network performance score; the grid network performance score is divided into five intervals: excellent, good, average, poor, and severe;

[0271] S35: Combine step S31, step S32, step S33 and step S34 to perform a comprehensive evaluation of audio and video perception and finally form a user perception evaluation;

[0272] The method for performing comprehensive evaluation of audio and video perception in step S35 specifically includes the following steps:

[0273] After obtaining three user speech perception scores through video perception evaluation, speech perception evaluation, network quality evaluation and text similarity respectively, different weights are set for the results obtained by the three methods according to experience, and the final user speech perception score is obtained by weighted average; the weight based on the video perception evaluation method is SW0, the weight based on the speech perception evaluation method is SW1, the weight based on the network quality evaluation method is SW2, and the weight based on the text similarity method is SW3; the final user speech perception comprehensive evaluation; the calculation formula is:

[0274] S ensemble =SW0*S0+SW1*S1+SW2*S2+SW3*S3;

[0275] In this embodiment, the weight of the video perception evaluation method is 0.2, the weight of the speech perception evaluation method is 0.3, the weight of the network quality evaluation method is 0.2, and the weight of the text similarity method is 0.3; the final user speech perception comprehensive evaluation; the calculation formula is:

[0276] S ensemble =0.2*S0+0.3*S1+0.2*S2+0.3*S3;

[0277] Among them: S0 is the scoring result based on the video perception evaluation method, S1 is the scoring result based on the speech perception evaluation method, S2 is the scoring result based on the network quality evaluation method, and S3 is the scoring result based on the text similarity method.

[0278] like Figure 4As shown, the system based on ViNR user audio and video perception evaluation includes a sending end module, a receiving end module and a perception evaluation module, the sending end module is connected to the receiving end module through a communication network, and the sending end module and the receiving end module are respectively communicated with the perception evaluation module; the sending end module includes a sending end audio and video data unit and a communication unit 1, and the communication unit 1 forms a data connection with the sending end audio and video data unit and the sending end audio and video data processing unit respectively; the receiving end module includes a receiving end audio and video data unit and a communication unit 2, and the communication unit 2 forms a data connection with the receiving end audio and video data unit and the receiving end audio and video data processing unit respectively; the perception evaluation module includes a sending end audio and video data processing unit, a data time series processing unit, a receiving end audio and video data processing unit, a video storage unit, a video similarity evaluation unit, an audio storage unit, an audio similarity evaluation unit, a time series storage unit The transmitting end module is connected to the transmitting end audio and video data processing unit of the perception evaluation module through a communication unit 1, and the receiving end audio and video data processing unit is connected to the receiving end audio and video data processing unit through a communication unit 2. The transmitting end module is connected to the transmitting end audio and video data processing unit of the perception evaluation module through a communication unit 1, and the receiving end module is connected to the receiving end audio and video data processing unit of the perception evaluation module through a communication unit 2.

[0279] By adopting the above technical scheme, the overall audio and video data of the sending end is processed, the video data of the sending end is stored in the video storage unit of the sending and receiving end, and the video time series data of the sending end is stored in the time series storage unit of the sending and receiving end; the audio data of the sending end is stored in the audio storage unit of the sending and receiving end, and the audio of the sending end is converted into text information of the sending end, and the text information of the sending end is stored in the text storage unit of the sending and receiving end, and the text information time series data of the sending end is stored in the time series storage unit of the sending and receiving end; and the network parameters and event information of the sending end are saved at the same time; the sending end sends the audio and video data of the sending end to the receiving end through the communication network, and after the receiving end receives the audio and video data, the overall audio and video data of the receiving end is processed, the video data of the receiving end is stored in the video storage unit of the sending and receiving end, and the video time series data of the receiving end is stored in the time series storage unit of the sending and receiving end; the audio data of the receiving end is stored in the audio storage unit of the sending and receiving end, and the audio of the receiving end is converted into text information of the receiving end, and the text information of the receiving end is stored in the text storage unit of the sending and receiving end Unit, stores the time series data of the text information of the receiving end into the time series storage unit of the sending and receiving end; saves the network parameters and event information of the receiving end at the same time; uses the video similarity method to evaluate the video similarity of the sending end video data and the receiving end video data, and displays them in real time; uses the audio similarity method to evaluate the audio similarity of the sending end audio data and the receiving end audio data, and displays them in real time; uses the text similarity method to evaluate the text similarity of the sending end audio text data and the receiving end audio text data, and displays them in real time; based on the network parameters and event information of the audio and video sending end and the audio and video receiving end, network quality evaluation is performed through network quality evaluation algorithms and methods; based on the time series data of the video, audio, and text of the sending end and the time series data of the video, audio, and text of the receiving end, time series evaluation is performed through time series alignment algorithms and methods; finally, video similarity, audio similarity, text similarity, network quality evaluation, and time series evaluation are combined to perform a comprehensive evaluation of audio and video perception, and finally form a user perception evaluation.

[0280] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A ViNR user audio and video perception evaluation method, characterized in that: The specific steps include: S1 audio and video data initiation: The sender initiates audio data and video data, and records and saves the time series data corresponding to the audio data and video data at the time of initiation; the audio data is processed and saved, and the network data of the sender is saved at the same time; S2 audio and video data reception: The receiving end receives audio data and video data, processes and saves the time series corresponding to the audio data and video data recorded when receiving; the audio data is processed and saved, and the network data of the receiving end is saved at the same time; S3 Evaluation and Analysis: Use similarity evaluation methods to evaluate various data from the sender and receiver, combine time series data for time series evaluation, and combine network data for network quality evaluation; finally, obtain a comprehensive evaluation of audio and video perception to form a user perception evaluation.

2. The ViNR user audio and video perception evaluation method according to claim 1 is characterized in that: The specific steps of step S3 are: S31: using a video similarity method, evaluating the similarity between the video data of the sending end in step S1 and the video data of the receiving end in step S2, and displaying the results in real time; S32: using a text similarity method, evaluating the similarity between the text information data converted from the audio data processed by the sending end in step S1 and the text information data converted from the audio data processed by the receiving end in step S2, and displaying them in real time; S33: Use voice information to establish a user perception evaluation model through a telecommunication psychology algorithm to evaluate the user's voice perception; S34: Evaluate the network quality based on the network parameters and event information of the sending end and the receiving end by using a network quality evaluation algorithm and method; S35: Combine steps S31, S32, S33 and S34 to perform a comprehensive evaluation of audio and video perception to finally form a user perception evaluation.

3. The ViNR user audio and video perception evaluation method according to claim 2 is characterized in that: The specific steps of using video similarity to evaluate video similarity in step S31 are: S311: The sending end divides the original video data according to the time stored in the time series data, so as to form an association relationship between each frame of the original image and a time point; S312: The receiving end collects the comparison video of the original video data through the communication network, and performs frame segmentation on the comparison video according to the time of the video frames stored in the time series to form a correspondence between each frame of the comparison image and the comparison time point; S313: Calculate the image similarity between each frame of the original image at each time point and each frame of the comparison image at the comparison time point, evaluate the video quality based on the image similarity calculation result, and finally output the result.

4. The ViNR user audio and video perception evaluation method according to claim 3 is characterized in that: The specific steps of using text similarity to evaluate file similarity in step S32 are: S321: generating a corresponding standard audio as comparison audio by mechanically reading the original audio data of the sending end, and then converting the comparison audio into original text; S322: The receiving end collects the comparison audio and converts it into a comparison text; S323: Calculate the text similarity between the original text and the comparison text using a text similarity algorithm, then perform function mapping conversion, and finally output the result.

5. The ViNR user audio and video perception evaluation method according to claim 4 is characterized in that: In step S33, the speech perception evaluation is performed by the telecommunication psychology algorithm, and various speech samples are evaluated by artificial perception to establish a user speech perception evaluation model. The specific steps of the speech perception evaluation are as follows: S331 data collection: collects voice audio files of senders and receivers under different network qualities and corresponding VoLTE / VONR network indicators; S332 data processing: Users listen to the voice audio files of the voice initiator and the voice receiver respectively, and vote on the quality of the audio according to their personal perception. A corresponding threshold is set according to the voting result. If users who exceed the threshold give a good score, the audio file is labeled 1; label 0 means that users who exceed the threshold give a bad score. Therefore, each VoLTE / VONR network indicator has its corresponding perception label. S333 Feature selection: Before building the classification model, the feature score in xgboost was used to screen the final variables; S334 Model establishment: Use multiple classification algorithms to train the training set based on the network indicators corresponding to the existing good and bad audio, and use the test set for verification to obtain the optimal classification model, and output the user perception model; S335 model prediction: perform user perception model prediction on the network indicators corresponding to the audio, and map the perception probability into a user perception score; in step S334, the score table for each audio file can be output through the established classification model.

6. The ViNR user audio and video perception evaluation method according to claim 5 is characterized in that: The specific steps of the network quality evaluation algorithm and method in step S34 are: S341 Data Collection: Collect network data including network parameters and event information, i.e. user GPS information, MR data and VoLTE / VONR data; S342 data processing: integrating and associating the data sources in step S322 at the grid level; S343 Data calculation and analysis: Before calculating the grid network performance index items, first calculate the basic network performance scores of each cell covering the grid. After obtaining the basic network performance scores of all cells covering the grid, use the algorithm to obtain the basic network performance score of the grid. S344 data analysis results: The service type is VoLTE / VONR service, and then select the time to be evaluated to obtain the grid network performance score.

7. The ViNR user audio and video perception evaluation method according to claim 6 is characterized in that: The basic network performance score of the cell in step S343 The scores of all KPI indicators are: The weighted sum is obtained to calculate the basic network performance score of each cell in the coverage grid That is, the KPI score of each cell is calculated using different algorithms according to the indicator attributes, specifically: If the indicator is smaller, the better: when When , the calculation formula is: in, KPI for all communities j The value of the 2.5%-97.5% quantile interval of the indicator, KPI for cell X j The range of the interval, where the numerator is the KPI in cell X j Cumulative distribution function of, with KPI as the denominator j The value corresponding to the cell with the largest cumulative distribution function; If the KPI of cell X j Less than When the left endpoint is , the calculation formula is: If the KPI of cell X j Greater than When the right endpoint is , the calculation formula is: If the indicator is bigger, the better: when When , the calculation formula is: If the KPI of cell X j Greater than The right endpoint of , the calculation formula is: If the KPI of cell X j Less than The left endpoint of , the calculation formula is: Finally, the basic network performance score of all cells in the coverage grid is obtained; In step S343, the basic network performance scores of all cells covering the grid are obtained through data calculation and analysis, and then the basic network performance scores of the grid are obtained by using an algorithm. The specific algorithm is: Among them, Grid x refers to a grid X, Refers to the set of all cells covering grid X; After obtaining the basic performance score of the grid based on the above algorithm logic, the MR data based on the user in the grid is added as the adjustment parameter Thus, the final basic network performance score of each grid is obtained, and the formula is: The range of this adjustment parameter is in is the value of grid X corresponding to the normalized value of the RSRP mean of all grids for 14 consecutive days, For each grid, the SINR mean value for a certain 14-day period is normalized to the min-max value corresponding to the SINR mean value of the grid; Normalization of min-max is also called deviation normalization, which is a linear transformation of the original data so that the result is mapped between 0 and 1. The conversion function is: Among them, max is the maximum value of the sample data, and min is the minimum value of the sample data; Finally, the final basic network performance score of the grid is obtained based on the basic network performance score of the grid and the adjustment parameters. The formula is: Finally, the basic network performance score The score is mapped to the interval (0, 100).

8. The ViNR user audio and video perception evaluation method according to claim 7, characterized in that: The method for performing comprehensive evaluation of audio and video perception in step S35 specifically includes the following steps: After obtaining three user speech perception scores through video perception evaluation, speech perception evaluation, network quality evaluation and text similarity respectively, different weights are set for the results obtained by the three methods, and the final user speech perception score is obtained by weighted average; the weight based on the video perception evaluation method is SW0, the weight based on the speech perception evaluation method is SW1, the weight based on the network quality evaluation method is SW2, and the weight based on the text similarity method is SW3; the final user speech perception comprehensive evaluation; the calculation formula is: <h2 style=";text-align:left;direction:ltr">S<h2 style=";text-align:left;direction:ltr"> ensemble <h2 style=";text-align:left;direction:ltr"> (SW0*S0+SW1*S1+SW2*S2+SW3*S3) Among them: S0 is the scoring result based on the video perception evaluation method, S1 is the scoring result based on the speech perception evaluation method, S2 is the scoring result based on the network quality evaluation method, and S3 is the scoring result based on the text similarity method.

9. The ViNR user audio and video perception evaluation method according to claim 7, characterized in that: Four text similarity algorithms are used in step S323, and the text similarity algorithms in step S33 include four text similarity algorithms, namely: a statistical algorithm based on word frequency (TF), a Simhash text similarity algorithm, a text similarity algorithm based on a vector space model VSM, and a text similarity algorithm based on an LDA topic model; The specific steps of the statistical algorithm based on word frequency are as follows: S332-1-1: List each word of the standard text; S332-1-2: Count the frequency of each word in the standard text in the standard text and the comparison text respectively. s 、f c , where f c ≤f s , the redundant words in the comparison text are not counted in the statistics; S332-1-3: Text similarity result, the formula is: Similarity tf is the similarity of word frequency TF; The specific steps of the Simhash text similarity algorithm include: S332-2-1: Segment the text into words respectively, and take the features and weights of the first n words with the highest TF-IDF weights in the text: that is, a set of (feature: weight) with a length of n is obtained for each text; S332-2-2: Perform a common hash on the feature of the word to obtain a 64-bit binary, and obtain a set of (hash: weight) with a length of 20; S332-2-3: According to the corresponding position of the binary number hash obtained in step S332-2-2, whether it is 1 or 0, take positive weight or negative weight for the corresponding position; S332-2-4: perform column vector addition on the n lists in step S332-2-3 to obtain a list; S332-2-5: judge each value in the list obtained in step S332-2-4, take 0 when it is a negative value, and take 1 when it is a positive value; S332-2-6: calculate the similarity, take XOR on the simhash of the two texts, if it is 1, it means they are different, and 0 means they are the same; the length of 1 divided by the overall length is the difference, and 1 minus the difference is the text similarity simhash ; The specific steps of the text similarity algorithm based on the vector space model VSM include: S332-3-1: Assume that a speech text has four feature items a, b, c, and d, then the speech text is represented by D(a, b, c, d); S332-3-2: For other texts to be compared with the speech text, the order of the feature items is also followed; for a text containing n feature items, a certain weight is given to each feature item to indicate its importance, that is, D = D (T1, W1; T2, W2; ..., T n , W n ) is abbreviated as D = D (W1, W2, ..., W n ), called the weight vector representation of text D; where Wk is the weight of Tk, 1<=k<=N; S332-3-3: In the vector space model, the content relevance Sim(D1, D2) between text D1 and text D2 is expressed by the cosine value of the angle between the vectors, and the formula is: The specific steps of the text similarity algorithm based on the LDA topic model include: firstly, using the LDA model to model the text set, that is, using the statistical characteristics of the text, mapping the text corpus to each topic space, mining the relationship between different topics and words hidden in the text, obtaining the topic distribution of the text, and then calculating the text similarity matrix through the topic distribution; The process of generating text by the LDA topic probability model is as follows: S332-4-1: For topic z, a word multinomial distribution vector on the topic is obtained according to the Dirichlet distribution Dir(β) S332-4-2: Obtain the number of words N in the text according to Poisson distribution P; S332-4-3: Obtain a topic distribution probability vector θ of the text according to Dirichlet distribution Dir(α); S332-4-4: For each word Wn in the N words of the text: S332-4-5: Randomly select a topic z from the multinomial distribution Multinomial(θ) of θ; S332-4-6: Select a word as Wn from the multinomial conditional probability distribution Multinomial(Φ) of topic z; Since the topic distribution of a text is a simple mapping of the text vector space, in the case of text topic representation, the similarity between two texts is calculated by calculating the corresponding topic probability distribution; since the topic is a mixed distribution of word vectors, the Kullback-Leibler relative entropy distance, denoted as KL distance, is used as the similarity metric. The calculation formula of KL distance is: Where D KL (p, q) represents the information loss when the probability distribution Q is used to fit the true distribution P, where P represents the true distribution and q represents the fitted distribution of P.

10. The ViNR user audio and video perception evaluation method according to claims 1-9, characterized in that: The system based on ViNR user audio and video perception evaluation includes a sending end module, a receiving end module and a perception evaluation module, wherein the sending end module is connected to the receiving end module via a communication network, and the sending end module and the receiving end module are respectively communicated with the perception evaluation module; the sending end module includes a sending end audio and video data unit and a communication unit 1, and the communication unit 1 forms a data connection with the sending end audio and video data unit and the sending end audio and video data processing unit respectively; the receiving end module includes a receiving end audio and video data unit and a communication unit 2, and the communication unit 2 forms a data connection with the receiving end audio and video data unit and the receiving end audio and video data processing unit respectively; the perception evaluation module includes a sending end audio and video data processing unit, a data time series processing unit, a receiving end audio and video data processing unit, a video storage unit, a video similarity evaluation unit, an audio storage unit, an audio similarity evaluation unit, a time series storage unit, A time series evaluation unit, a text storage unit, a text similarity evaluation unit, a signal storage unit, a network quality evaluation unit and a user perception evaluation unit, the sending end audio and video data processing unit and the receiving end audio and video data processing unit are both connected to the data time series processing unit, the video storage unit, the audio storage unit, the text storage unit and the signal storage unit in communication, the video storage unit is electrically connected to the video similarity evaluation unit, the audio storage unit is electrically connected to the audio similarity evaluation unit, the data time series processing unit is electrically connected to the time series storage unit, the time series storage unit is electrically connected to the time series evaluation unit, the text storage unit is electrically connected to the text similarity evaluation unit, the signal storage unit is electrically connected to the network quality evaluation unit, the video similarity evaluation unit, the audio similarity evaluation unit, the time series evaluation unit, the text similarity evaluation unit and the network quality evaluation unit are all electrically connected to the user perception evaluation unit. The sending end module is connected to the sending end audio and video data processing unit of the perception evaluation module through the communication unit 1, and the receiving end module is connected to the receiving end audio and video data processing unit of the perception evaluation module through the communication unit 2.

Images