A performance evaluation method, device and equipment of an intelligent connected vehicle and a medium

Abstract

The present application relates to the field of vehicle communication, and relates to a performance evaluation method and device for intelligent networked vehicles, equipment and medium. The present application obtains network performance evaluation instructions for intelligent networked vehicles, specifies performance evaluation types and target evaluation strategies, which makes the test no longer limited to the single isolated road test data mode in the past, and can obtain data from multiple aspects according to different evaluation types. Secondly, the response instruction uses the performance evaluation type to obtain the target network performance data associated with the measured equipment and the equipment operation log, comprehensively covering various situation records in the equipment operation process, so as to realize comprehensive quantitative recording of network related indicators and change the previous situation of lacking comprehensive quantitative recording. Furthermore, in the process of evaluating the target network performance data according to the target evaluation strategy to obtain the performance evaluation result, comparison and analysis of data under different conditions will be involved, which makes up for the lack of horizontal and vertical data analysis in the past.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of vehicle communication, in particular to a performance evaluation method and device for intelligent connected vehicles, equipment and medium. BACKGROUND

[0002] As a basic technology of intelligent connected vehicles, the importance of network communication system cannot be underestimated. In the field of intelligent connected vehicle cabin navigation, it ensures the real-time transmission of map data and the accurate issuance of navigation instructions, allowing drivers to accurately obtain route guidance. In the field of automatic driving, network communication system is even more critical. Sensor data of vehicles need to be quickly and accurately transmitted and interacted through it, so that the automatic driving system can make timely decisions based on this information to ensure driving safety and smoothness.

[0003] However, at present, there are many deficiencies in the testing and analysis of intelligent connected vehicles. Most of the work only analyzes isolated road test data. Researchers often do not comprehensively quantify network-related indicators, which leads to many problems. For example, due to the lack of cross-comparison analysis, researchers cannot clearly understand the differences in network performance of different vehicles or the same vehicle in different environments. The lack of longitudinal historical data analysis also makes it difficult for them to grasp the trend of network performance over time. More seriously, there is no automated testing and analysis method combining large-scale batch testing with historical big data in the entire industry, which greatly limits the optimization and improvement of intelligent connected vehicle network communication systems and is not conducive to their further development. SUMMARY

[0004] Therefore, the embodiments of the present application provide a performance evaluation method and device for intelligent connected vehicles, equipment and medium to solve the problem that existing testing and analysis are mostly limited to isolated road test data, lack of comprehensive quantitative records of network-related indicators, lack of cross and longitudinal data analysis, and lack of automated testing and analysis method combining large-scale batch testing with historical big data.

[0005] In a first aspect, the embodiments of the present application provide a performance evaluation method for intelligent connected vehicles, which comprises:

[0006] obtaining a network performance evaluation instruction for an intelligent connected vehicle, wherein the network performance evaluation instruction comprises a performance evaluation type and a target evaluation strategy;

[0007] in response to the network performance evaluation instruction, obtaining target network performance data associated with a measured device in the intelligent connected vehicle using the performance evaluation type, and obtaining a device operation log corresponding to the measured device;

[0008] According to the target evaluation strategy, the target network performance data is evaluated to obtain a performance evaluation result corresponding to the device under test, and a corresponding processing operation is performed on the device running log according to the performance evaluation result.

[0009] Further, the target network performance data associated with the device under test in the intelligent connected vehicle is obtained by using the performance evaluation type, including:

[0010] If the performance evaluation type is a real-time network basic coverage performance evaluation type, first network basic performance data of the device under test and second network basic performance data of a reference machine are obtained, data smoothing processing is performed on the first network basic performance data and the second network basic performance data according to a preset processing rule, and the processed first network basic performance data and the processed second network basic performance data are taken as the target network performance data.

[0011] If the performance evaluation type is a real-time network service performance evaluation type, first network service performance data of the device under test and second network service performance data of a reference machine are obtained, data smoothing processing is performed on the first network service performance data and the second network service performance data according to a preset processing rule, and the processed first network service performance data and the processed second network service performance data are taken as the target network performance data.

[0012] If the performance evaluation type is a network performance comprehensive evaluation type, first network performance data of the device under test and second network performance data of a reference machine are obtained, first space-time information corresponding to the first network performance data and second space-time information corresponding to the second network performance data are determined, first historical network performance data associated with the first space-time information and second historical network performance data matched with the second space-time information are obtained, and the first network performance data, the second network performance data, the first historical network performance data and the second historical network performance data are taken as the target network performance data.

[0013] If the performance evaluation type is a batch network performance evaluation type, network basic performance data of the device under test contained in the current batch is obtained, and the network basic performance data of the device under test contained in the current batch is taken as the target network performance data.

[0014] Further, the first historical network performance data associated with the first space-time information and the second historical network performance data matched with the second space-time information are obtained, including:

[0015] obtain a network map data model, wherein the network map data model comprises historical network performance data corresponding to different historical positions and different historical times of the device under test;

[0016] extract time information and spatial position information from the first spatio-temporal information;

[0017] obtain a historical relative position matching the spatial position information from the network map data model, and obtain historical network performance data corresponding to different historical times associated with the historical relative position;

[0018] obtain first historical network performance data matching the time information from the historical network performance data corresponding to different historical times associated with the first historical relative position, and obtain second historical network performance data matching the second spatio-temporal information from the historical network performance data set corresponding to the reference device;

[0019] Further, the obtaining a historical relative position matching the spatial position information from the network map data model comprises:

[0020] obtaining potential historical position information adjacent to the spatial position information from the network map data model;

[0021] calculating a position difference based on the spatial position information and the potential historical position information, and taking the potential historical position information with a position difference less than a preset distance difference value as candidate historical position information;

[0022] calculating a first speed value according to a distance change between the spatial position information and the candidate historical position information and a first time difference, wherein the first time difference is calculated based on the time information and historical time information corresponding to the candidate historical position information;

[0023] calculating a second speed value according to a distance change between the candidate historical position information at different historical times and a second time difference, wherein the second time difference is calculated based on different historical time information corresponding to the candidate historical position information;

[0024] calculating a speed difference value between the first speed value and the second speed value, and determining the historical relative position by using the candidate historical position information with a speed difference value less than a preset speed difference value.

[0025] Further, the method further comprises:

[0026] performing correlation analysis on the first network performance data and historical network performance data in the network map data model to obtain spatio-temporal features and a plurality of correlation data between the first network performance data and the historical network performance data.

[0027] updating the correlation data to a network map data model according to the spatio-temporal characteristics;

[0028] fusing the correlation data between the first network performance data and the historical network performance data according to a preset algorithm in the network map data model, to obtain an optimized network map data model.

[0029] Further, the evaluating the target network performance data according to the target evaluation strategy to obtain a performance evaluation result corresponding to the measured device comprises:

[0030] if the performance evaluation type is a real-time network basic coverage performance evaluation type, obtaining first index data corresponding to a preset index from the first network basic performance data, and obtaining second index data corresponding to the preset index from the second network basic data;

[0031] comparing the first index data and the second index data associated with each preset index;

[0032] if the first index data associated with at least one preset index is lower than the second index data, determining that the performance evaluation result is that the basic performance of the measured device is lower than the basic performance of the reference device; or, if the first index data associated with each preset index is higher than the second index data, determining that the performance evaluation result is that the basic performance of the measured device is higher than the basic performance of the reference device.

[0033] Further, the evaluating the target network performance data according to the target evaluation strategy to obtain a performance evaluation result corresponding to the measured device comprises:

[0034] if the performance evaluation type is a real-time network service performance evaluation type, obtaining third index data corresponding to a preset index from the first network service performance data, and obtaining fourth index data corresponding to the preset index from the second network service data;

[0035] comparing the third index data and the fourth index data associated with each preset index;

[0036] if the third index data associated with at least one preset index is lower than the fourth index data, determining that the performance evaluation result is that the service performance of the measured device is lower than the service performance of the reference device; or, if the third index data associated with each preset index is higher than the fourth index data, determining that the performance evaluation result is that the service performance of the measured device is higher than the service performance of the reference device.

[0037] Further, the evaluating the target network performance data according to the target evaluation strategy to obtain the performance evaluation result corresponding to the DUT comprises:

[0038] If the performance evaluation type is a network performance comprehensive evaluation type, the current performance of the DUT is obtained by comparing the first network performance data and the second network performance data;

[0039] The historical performance of the DUT is obtained by comparing the first historical network performance data and the second historical network performance data;

[0040] The change trend of the network performance of the DUT over time is obtained by comparing the first network performance data and the first historical network performance data;

[0041] The performance difference of the DUT is obtained by analyzing the current performance and the historical performance;

[0042] The performance evaluation result is generated based on the performance difference of the DUT and the change trend.

[0043] Further, the evaluating the target network performance data according to the target evaluation strategy to obtain the performance evaluation result corresponding to the DUT comprises:

[0044] If the performance evaluation type is a batch network performance evaluation type, the network basic performance data of the DUT in the current batch is processed according to a preset processing rule to obtain the processed network basic performance data of the DUT in the current batch;

[0045] The space-time state information corresponding to the network basic performance data of the DUT in the current batch is obtained;

[0046] The associated network basic performance data of each DUT in the current batch is obtained by using the space-time state information;

[0047] The performance evaluation result corresponding to the DUT is obtained by comparing the network basic performance data of each DUT in the current batch with the associated network basic performance data;

[0048] Further, the obtaining the associated network basic performance data of each DUT in the current batch by using the space-time state information comprises:

[0049] The network map data model is obtained, wherein the network map data model comprises historical network performance data corresponding to different historical positions of the DUT and different historical times;

[0050] obtain a historical network performance data set corresponding to the tested device from the network map data model;

[0051] filter, from the historical network performance data set, associated network basic performance data corresponding to each tested device in the current batch by using the spatio-temporal state information.

[0052] In a second aspect, an embodiment of the present application provides a performance evaluation device for an intelligent connected vehicle, the device comprising:

[0053] an obtaining module configured to obtain a network performance evaluation instruction for the intelligent connected vehicle, wherein the network performance evaluation instruction comprises a performance evaluation type and a target evaluation strategy;

[0054] a responding module configured to respond to the network performance evaluation instruction, obtain target network performance data associated with a tested device in the intelligent connected vehicle by using the performance evaluation type, and obtain a device running log corresponding to the tested device;

[0055] an evaluation module configured to evaluate the target network performance data according to the target evaluation strategy, obtain a performance evaluation result corresponding to the tested device, and perform a corresponding processing operation on the device running log according to the performance evaluation result.

[0056] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory and a processor, which are communicatively connected with each other, and the memory stores computer instructions, and the processor executes the computer instructions to perform the method of the first aspect or any of the corresponding embodiments thereof.

[0057] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores computer instructions, and the computer instructions are used to make a computer perform the method of the first aspect or any of the corresponding embodiments thereof.

[0058] The method provided by the embodiments of the present application obtains the network performance evaluation instruction for the intelligent connected vehicle, clearly defines the performance evaluation type and the target evaluation strategy, which makes the test no longer limited to the single isolated road test data mode in the past, and enables data to be obtained from multiple aspects according to different evaluation types. Secondly, the responding instruction obtains the target network performance data associated with the tested device and the device running log by using the performance evaluation type, comprehensively covers various situation records in the device running process, thereby realizing comprehensive quantitative recording of network related indexes, and changing the situation of lacking comprehensive quantitative recording in the past. Furthermore, in the process of evaluating the target network performance data according to the target evaluation strategy to obtain the performance evaluation result, comparison and analysis of data under different conditions are involved, which makes up for the lack of horizontal and vertical data analysis in the past.

[0059] By obtaining the first index data and the second index data according to the preset indexes from the first network basic performance data and the second network basic data, the evaluation can be focused on the key evaluation elements, and the evaluation is more targeted and accurate, and the interference of irrelevant data is avoided. Secondly, by comparing the two groups of data associated with each preset index, the differences between the measured device and the reference machine in each key network basic performance index can be clearly shown. This one-by-one comparison method can deeply explore the differences between the performances of the two. Finally, according to the comparison result, the high-low relationship between the basic performance of the measured device and the reference machine is determined. Whether there is at least one index data lower indicating that the performance of the measured device is poor, or all index data is high indicating that the performance is excellent, a clear conclusion can be provided for network basic performance evaluation, which helps to quickly and accurately judge whether the measured device meets the standard or has advantages and disadvantages in network basic performance, and then guide the subsequent optimization, improvement or selection and other related work.

[0060] When the performance evaluation type is the real-time network service performance evaluation type, the third index data and the fourth index data are obtained from the first network service performance data and the second network service data according to the preset indexes. This can accurately locate the key service performance index, so that the evaluation is focused on the factors that have an important influence on the actual network service running status, effectively eliminates irrelevant information interference, and makes the evaluation more targeted. Secondly, the third index data and the fourth index data associated with each preset index are compared one by one, which can clearly and carefully present the specific differences between the measured device and the reference machine in each key service performance index, so as to comprehensively and deeply understand the different performances of the two in network service performance. Thirdly, according to the comparison result, the high-low relationship between the service performance of the measured device and the reference machine is determined. If there is at least one preset index data lower than the other party, it can be clearly known that the service performance of the measured device is poor, otherwise if all preset index data is higher than the other party, it indicates that the service performance is better. In this way, an exact conclusion can be given for network service performance evaluation, which helps to quickly and accurately judge the advantages and disadvantages of the measured device in network service performance, and then provides a strong basis for subsequent optimization adjustment, equipment selection and other related work for service performance.

[0061] When the performance evaluation type is the network performance comprehensive evaluation type, the current performance situation of the device under test is obtained by comparing the first network performance data and the second network performance data, which can intuitively present the performance of the device under test relative to the reference machine in various network performance indicators at the current moment, clearly grasp its running condition in the current network environment, including the instant situation such as network speed, stability and the like. Secondly, the historical performance situation of the device under test is obtained by comparing the first historical network performance data and the second historical network performance data, which makes it possible to deeply understand the past network performance of the device under test from the time dimension, and to understand its performance trend over time, such as whether there is a gradual decline or improvement in performance, etc., providing an important historical reference basis for comprehensive evaluation. Finally, the performance evaluation result corresponding to the device under test is obtained by comprehensively analyzing the current performance situation and the historical performance situation, and this comprehensive evaluation method can more accurately judge the overall network performance of the device under test. It not only considers the current performance, but also takes into account the historical development trend, so that potential performance problems can be found more accurately, providing a very reliable and comprehensive decision basis for subsequent targeted network optimization, device upgrade or adjustment, etc., which helps to better guarantee the efficient and stable operation of the network system. BRIEF DESCRIPTION OF DRAWINGS

[0062] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the specific embodiments or prior art description. Obviously, the drawings described below are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0063] Figure 1 is a flowchart of a performance evaluation method of an intelligent connected vehicle according to some embodiments of the present application;

[0064] Figure 2 is a schematic diagram of a test system according to some embodiments of the present application;

[0065] Figure 3 is a flowchart of another performance evaluation method of an intelligent connected vehicle according to some embodiments of the present application;

[0066] Figure 4 is a flowchart of another performance evaluation method of an intelligent connected vehicle according to some embodiments of the present application;

[0067] Figure 5 is a flowchart of another performance evaluation method of an intelligent connected vehicle according to some embodiments of the present application;

[0068] Figure 6is a flowchart of a performance evaluation method of another intelligent connected vehicle according to some embodiments of the present application;

[0069] Figure 7 is a structural block diagram of a performance evaluation device of an intelligent connected vehicle according to an embodiment of the present application;

[0070] Figure 8 is a hardware structure schematic diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0071] To make the objects, technical solutions and advantages of embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0072] According to an embodiment of the present application, a performance evaluation method, device, equipment and medium of an intelligent connected vehicle are provided. It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a group of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that here.

[0073] In the present embodiment, a performance evaluation method of an intelligent connected vehicle is provided, Figure 1 is a flowchart of a performance evaluation method of an intelligent connected vehicle according to an embodiment of the present application, as Figure 1 shown, the flowchart includes the following steps:

[0074] In step S101, a network performance evaluation instruction for an intelligent connected vehicle is obtained, wherein the network performance evaluation instruction includes a performance evaluation type and a target evaluation strategy.

[0075] In the present embodiment, as Figure 2 shown, the measured device and the reference machine are both connected with the control unit, and this connection mode has multiple possibilities. It can be through a physical connection mode such as USB, and data transmission and control functions can be realized by means of tools such as adb; it can also be based on virtual connection, such as wireless personal area network (WPAN), local area network (LAN), Internet, etc., and corresponding operations can also be completed by means of tools such as adb, apk. This provides multiple choices for testing under different environments and needs, and the appropriate connection approach can be selected flexibly according to the actual situation.

[0076] At the same time, the control unit sends the reference device and the device under test the same action target instructions. This means that the tasks assigned to both are the same, so that their execution can be compared under the same conditions. Then, the reference device and the device under test execute the instructions and return the results to the control unit, so that the control unit can obtain the data of the completion of the same task by both. When the adb tool based on USB physical connection is used, the control, query and other instructions are accurately sent to the device under test and the reference device according to the ID number of the device. The specific instruction format is "adb-s <device under test or reference device ID number> shell <encapsulated instruction>". In this way, the specific device can be accurately located and specific instruction content can be sent to it, so as to ensure that the instructions can accurately reach the target device and be executed.

[0077] The control unit also obtains the time-stamped latitude and longitude information of the reference device or the device under test. This latitude and longitude information is very important, which is equivalent to marking the network performance data with time and space. By associating the network performance data with the time and space represented by the obtained time stamp and latitude and longitude information, the results of the network test can be traceable. That is, when it is necessary to view or analyze the network performance at a specific time and a specific place in the future, the corresponding test results can be easily found. At the same time, this makes the test results easy to repeat (retest) because the test environment can be accurately restored according to the previously recorded time and space conditions; and easy to compare, such as comparing the network performance differences between different times, different places or different devices, so as to more comprehensively and deeply analyze the network performance status.

[0078] In the embodiments of the present application, when the network performance of the intelligent connected vehicle is evaluated, the corresponding evaluation instruction is obtained, which covers the performance evaluation type and the target evaluation strategy. The performance evaluation type includes real-time network basic performance evaluation based on reference golden machine, i.e., comparing the network basic performance data such as RSSI, RSRP, SNR and RSRQ of the measured device and the reference golden machine at the same time point, combining data processing and the network, system and application logs of the measured device for monitoring; real-time network service performance evaluation based on reference golden machine, i.e., comparing the service performance data such as ping, upload, download and video playback and time and location information after processing by initiating service instructions to the measured device and the reference golden machine; historical-real-time comprehensive evaluation of the network performance (basic and service) of the reference golden machine and the measured device, which can compare the network basic and service performance data in real time and also can retrieve the historical data with time stamp for comprehensive comparison on demand; self-network basic performance evaluation of the measured device based on historical data, i.e., comparing the current network basic performance data of the measured device with its own historical record by using a large historical database; large-scale network basic performance evaluation, i.e., comparing the network basic performance data and time and location information obtained by using wireless network remote control with the associated data for a large number of measured devices. The target evaluation strategy is to combine log management after data acquisition, processing and comparison according to different types, such as deciding log storage or deletion, triggering alarm and the like according to the comparison results, so as to effectively evaluate the network performance of the intelligent connected vehicle.

[0079] In step S102, in response to the network performance evaluation instruction, the target network performance data associated with the measured device in the intelligent connected vehicle is obtained by using the performance evaluation type, and the device running log corresponding to the measured device is obtained.

[0080] In the embodiments of the present application, the target network performance data associated with the measured device in the intelligent connected vehicle is obtained by using the performance evaluation type, including the following cases:

[0081] ① If the performance evaluation type is real-time network basic coverage performance evaluation type, the first network basic performance data of the measured device and the second network basic performance data of the reference golden machine are obtained, the first network basic performance data and the second network basic performance data are subjected to data smoothing processing according to the preset processing rule, and the processed first network basic performance data and the processed second network basic performance data are taken as the target network performance data.

[0082] Specifically, when the performance evaluation type is determined to be the real-time network infrastructure performance evaluation type, a series of operation procedures are taken. First, the relevant network infrastructure performance data of the device under test, referred to as first network infrastructure performance data, and the network infrastructure performance data of the reference machine, referred to as second network infrastructure performance data, are obtained. Then, the two sets of obtained network infrastructure performance data, i.e., the first network infrastructure performance data and the second network infrastructure performance data, are respectively subjected to data smoothing processing according to a pre-set processing rule. Finally, the two sets of processed data, i.e., the processed first network infrastructure performance data and the processed second network infrastructure performance data, are determined as target network performance data for subsequent further analysis and comparison, so as to carry out related evaluation work such as judging the difference between the device under test and the reference machine in terms of network infrastructure performance based on the target network performance data.

[0083] The data smoothing processing may be implemented in the form of moving average, specifically, a time period of 10 seconds may be selected for moving average calculation of the relevant network infrastructure performance data within the time period, and the data in each time period is processed in turn by continuously sliding the time period, so that the data fluctuation is smoothed and the overall trend is highlighted. Alternatively, a position distance difference of 100 m may be selected for moving average, i.e., the data within each position distance range of 100 m apart is subjected to corresponding average calculation, and the operation is repeated with the position changing in turn, so as to reduce the dispersion degree of the data and make the data smoother.

[0084] If the performance evaluation type is the real-time network service performance evaluation type, the first network service performance data of the device under test and the second network service performance data of the reference machine are obtained, the first network service performance data and the second network service performance data are subjected to data smoothing processing according to a pre-set processing rule, and the processed first network service performance data and the processed second network service performance data are used as target network performance data.

[0085] Specifically, when the performance evaluation type is determined to be the real-time network service performance evaluation type, a series of operation processes are performed. Specifically, the first thing to do is to obtain the first network service performance data corresponding to the device under test, which involves indicators such as ping, upload, download, video playback, etc. that can reflect the actual running status of network service; at the same time, the second network service performance data of the reference machine is also obtained. Then, according to the pre-set processing rule, the two sets of data, i.e. the first network service performance data and the second network service performance data, are subjected to data smoothing processing. Finally, the two sets of data obtained after processing, i.e. the processed first network service performance data and the processed second network service performance data, are determined as the target network performance data, so as to carry out subsequent related work such as comparative analysis based on these target network performance data, so as to evaluate the performance of the device under test relative to the reference machine in terms of network service performance.

[0086] The data smoothing processing is achieved by, for example, using a moving average method. Specifically, a time period of 10 seconds can be selected for moving average calculation of the relevant network basic performance data within the time period, and the data of each time period is processed in turn by continuously sliding the time period, so that the data fluctuation is smoothed and the overall trend is highlighted. Alternatively, a position distance difference of 100 m can be used as the basis for moving average, i.e. the data within each position distance range of 100 m apart is subjected to corresponding average calculation, and this operation is repeated with the position changing in turn, so as to reduce the dispersion of the data and make the data smoother.

[0087] ③If the performance evaluation type is the network performance comprehensive evaluation type, the first network performance data of the device under test and the second network performance data of the reference machine are obtained, the first spatio-temporal information corresponding to the first network performance data and the second spatio-temporal information corresponding to the second network performance data are determined, the first historical network performance data associated with the first spatio-temporal information and the second historical network performance data matching the second spatio-temporal information are obtained, and the first network performance data, the second network performance data, the first historical network performance data and the second historical network performance data are taken as the target network performance data.

[0088] Specifically, obtaining the first historical network performance data associated with the first spatio-temporal information and the second historical network performance data matching the second spatio-temporal information includes the following steps 1-7:

[0089] Step 1, obtaining a network map data model, wherein the network map data model includes historical network performance data corresponding to different historical positions and different historical times of the device under test.

[0090] Specifically, by accessing the existing network map data model through the corresponding data access interface. This model contains the historical network performance data corresponding to the different historical positions and different historical times of the device under test. For the collection of historical network performance data, it is possible to obtain by continuously monitoring the network basic performance data (RSSI / RSRP / SNR / RSRQ, etc.) and network service performance data (such as ping, upload, download, video playback, etc.) of the device under test at different times and positions in the past, and recording the corresponding space-time state information. These data are organized according to geographical map information, so as to accurately reflect the network performance status at different positions and times.

[0091] Step 2, extract time information and spatial position information from the first space-time information.

[0092] Specifically, after obtaining the first space-time information, analyze the structure and content of the information. Use specific data extraction algorithms or tools to separate the time information part and the spatial position information part from the first space-time information. For time information, specific date, time point, etc. information can be extracted. For spatial position information, it can be parsed according to different representation methods, for example, if it is latitude and longitude coordinates, the corresponding longitude and latitude values are extracted. Ensure that the extracted time information and spatial position information are accurate and correct for subsequent steps.

[0093] Step 3, obtain the historical relative position matched with the spatial position information from the network map data model, and obtain the historical network performance data corresponding to different historical times associated with the historical relative position.

[0094] Specifically, obtaining the historical relative position matched with the spatial position information from the network map data model includes:

[0095] ①Obtain the potential historical position information adjacent to the spatial position information from the network map data model.

[0096] According to the positioning information and the derivative of the difference between the two adjacent positions with respect to time, select according to a certain error. Potential historical position information selection: according to the formula a1=(N+H)cosbcosL, b1=(N+H)cosBsinL, c1=[N(1-e 2 )+H]sinB, where (L, B, H) are geodetic latitude, longitude and height, (a1, b1, c1) are the positions calculated to the spatial rectangular coordinate system, N is the radius of the prime vertical circle of the ellipsoid, e is the first eccentricity of the ellipsoid, a and b are the long and short semi-axes of the earth.

[0097] ii. Calculate the position difference based on the spatial position information and the potential historical position information, and take the potential historical position information with a position difference less than a preset distance difference value as the candidate historical position information.

[0098] In the spatial rectangular coordinate system, take (a1, b1, c1), (a2, b2, c2),..., (a n ,b n ,c n ) as the position center of the measured device currently generated according to time. Take (A1, B1, C1), (A2, B2, C2),..., (A n ,B n ,C n ) as the potential historical position information, and so on to calculate the position difference, according to the distance formula between two points:

[0099] The relative position relationship between the current position and the historical position can be judged, and when d is less than the preset distance difference value, the historical position is considered to be preliminarily selected. The selected historical position is taken as the candidate historical position information.

[0100] iii. Calculate the first speed value according to the distance change between the spatial position information and the candidate historical position information and the first time difference, wherein the first time difference is calculated from the time information and the historical time information corresponding to the candidate historical position information.

[0101] The speed formula of the first speed value for the candidate historical position information is: t1, t2 are the current time difference.

[0102] iv. Calculate the second speed according to the distance change between the candidate historical position information at different historical time and the second time difference, wherein the second time difference is calculated from the candidate historical position information corresponding to different historical time information;

[0103] The speed formula of the second speed value is: T1, T2 are the historical time difference.

[0104] v. Calculate the speed difference value between the first speed value and the second speed value, and determine the historical relative position by using the candidate historical position information with a speed difference value less than a preset speed difference value.

[0105] Specifically, ΔV = v - V, when ΔV meets the set value, it is considered that the network information of the historical position is effective. That is, the speed change of each historical relative position is analyzed, and the historical relative position with a speed difference value meeting the first speed difference set requirement is selected out.

[0106] By obtaining the potential historical location information adjacent to the spatial location information from the network map data model, the existing historical data can be fully utilized to assist in determining the related information of the current location. Then, by calculating the location difference and screening out the potential historical location information with a location difference less than a preset distance difference value as the candidate historical location information, the analysis range can be reduced, and the processing efficiency and accuracy can be improved. Then, by calculating the first speed value according to the distance change and the time difference between the spatial location information and the candidate historical location information, and calculating the second speed value according to the distance change and the time difference between the candidate historical location information at different historical time points, the movement of the measured device at different time periods can be more comprehensively understood. Finally, by calculating the speed difference value and using the candidate historical location information with a speed difference value less than a preset speed difference value to determine the historical relative position, the historical position related to the current position can be more accurately determined, which provides a more reliable basis for subsequent network performance analysis and optimization of the network map data model, so as to better reflect the network performance status at different times and places, and improve the accuracy and practicability of the network map data model.

[0107] Step 4: obtaining the first historical network performance data matching the time information from the historical network performance data corresponding to different historical times associated with the first historical relative position, and obtaining the second historical network performance data matching the second spatio-temporal information from the historical network performance data set corresponding to the reference terminal.

[0108] The storage location of the first historical relative position in the network map data model is determined. From the set of historical network performance data corresponding to different historical times associated with the position, the matching search is performed according to the currently obtained time information. By comparing the closeness of each historical time and the current time information, the historical network performance data closest to the current time is selected as the first historical network performance data.

[0109] For obtaining the second historical network performance data from the historical network performance data set corresponding to the reference terminal, the time and space information in the second spatio-temporal information are first analyzed and extracted. Then, in the historical network performance data set of the reference terminal, matching is performed according to the time and space dimensions. In the time dimension, the historical record closest to the time in the second spatio-temporal information is found; in the space dimension, it is ensured that the position corresponding to the record has a certain correlation with the spatial position in the second spatio-temporal information. By comprehensively matching the time and space, the historical network performance data matching the second spatio-temporal information is determined as the second historical network performance data.

[0110] If the performance evaluation type is a batch network performance evaluation type, the network basic performance data of the measured device contained in the current batch is obtained, and the network basic performance data of the measured device contained in the current batch is taken as the target network performance data.

[0111] Specifically, when the performance evaluation type is set to the batch network performance evaluation type, the operation focuses on network performance evaluation for a plurality of devices under test in a batch. In this case, there is no need for comparison with a reference machine or introduction of historical data and other complex operations, but the network basic performance data of all devices under test in the batch currently being processed is directly extracted, which may include relevant indicators reflecting network basic running conditions such as RSSI, RSRP, and the like, and then the network basic performance data of all devices under test in the batch is directly taken as target network performance data, so that the devices under test in the batch are subjected to concentrated network basic performance analysis, evaluation and the like based on the data, to meet the demand for fast and overall grasp of network performance in a large-scale test or batch processing scenario.

[0112] In the embodiment of the present application, the method further comprises: performing correlation analysis on the first network performance data and historical network performance data in the network map data model to obtain spatio-temporal features and a plurality of correlation data between the first network performance data and the historical network performance data; updating the correlation data to the network map data model according to the spatio-temporal features; and fusing the plurality of correlation data between the first network performance data and the historical network performance data according to a preset algorithm in the network map data model to obtain an optimized network map data model.

[0113] First, the first network performance data corresponding to the device under test and the historical network performance data in the network map data model are subjected to correlation analysis. In this process, by comparing information in time, space and other dimensions, spatio-temporal features are extracted, and a plurality of correlation data between the first network performance data and the historical network performance data are determined, such as similarity of network performance parameters at a specific time and location, change trend and the like.

[0114] Next, according to the extracted spatio-temporal features, the correlation data is updated to the network map data model. This means that the corresponding time and location nodes are found in the network map data model, and the new correlation data is added to enrich the data content of the model.

[0115] Finally, according to the preset algorithm in the network map data model, the plurality of correlation data between the first network performance data and the historical network performance data are fused. Weighted average and other methods can be used to determine the weight according to the importance, timeliness and other factors of the data, and different data are calculated comprehensively to obtain an optimized network map data model, so that it can more accurately reflect the network performance conditions at different times and locations.

[0116] Step S103, evaluate the target network performance data according to the target evaluation strategy, obtain the performance evaluation result corresponding to the device under test, and perform corresponding processing operation on the device running log according to the performance evaluation result.

[0117] In the embodiments of the present application, the target network performance data is evaluated according to the target evaluation strategy, and the performance evaluation result corresponding to the device under test is obtained, as shown in Figure 3

[0118] Step A1, if the performance evaluation type is a real-time network basic coverage performance evaluation type, obtaining first index data corresponding to the preset index from the first network basic performance data, and obtaining second index data corresponding to the preset index from the second network basic data.

[0119] Specifically, when the performance evaluation type is determined as the real-time network basic coverage performance evaluation type, the relevant data obtained is first to be determined. In this context, there are first network basic performance data (these data should be obtained from the device under test before, containing various types of index information such as RSSI, RSRP, SNR, RSRQ, which can reflect the network basic performance) and second network basic performance data (which is the same type of network basic performance index data obtained from the reference machine).

[0120] Then, for those pre-set key indicators (i.e. preset index, such as the indicators of RSSI and RSRP which may be particularly concerned as the key evaluation basis) used for evaluation, the specific data corresponding to these preset indicators is accurately extracted from the first network basic performance data, which is defined as the first index data. Similarly, the corresponding second index data is also accurately obtained from the second network basic performance data according to the requirements of the preset index. In this way, the operation of extracting the corresponding specific index data from the two groups of network basic performance data according to the preset index is completed, which prepares for the subsequent comparison and analysis.

[0121] Step A2, compare the first index data and the second index data associated with each preset index.

[0122] ​Specifically, for each preset index, the first index data obtained from the to-be-tested device is compared with the second index data obtained from the reference device one by one. For example, if the preset index is RSSI, the first index data about RSSI extracted from the first network basic performance data is compared with the second index data about RSSI extracted from the second network basic performance data to check the size relationship between the two values; the same comparison is also performed on other preset indexes such as RSRP, SNR, RSRQ, and the like, and detailed comparison and analysis are performed on two sets of data corresponding to each preset index in sequence, so as to determine the performance difference between the to-be-tested device and the reference device on each key index.

[0123] Step A3, if the first index data associated with at least one preset index is lower than the second index data, it is determined that the performance evaluation result is that the basic performance of the to-be-tested device is lower than the basic performance of the reference device; or, if the first index data associated with the preset index is all higher than the second index data, it is determined that the performance evaluation result is that the basic performance of the to-be-tested device is higher than the basic performance of the reference device.

[0124] Specifically, if it is found that, among all the compared preset indexes, there is at least one preset index associated with the first index data lower than the corresponding second index data, it means that the to-be-tested device performs worse than the reference device on these key indexes, and thus it can be determined that the performance evaluation result is that the basic performance of the to-be-tested device is lower than the basic performance of the reference device.

[0125] On the contrary, if it is found that, after comparison one by one, the first index data associated with all the preset indexes is all higher than the corresponding second index data, it means that the to-be-tested device performs better than the reference device on each key index, and thus it can be determined that the performance evaluation result is that the basic performance of the to-be-tested device is higher than the basic performance of the reference device. In this way, the advantages and disadvantages of the to-be-tested device and the reference device in real-time network basic performance are accurately determined according to the comparison of the preset index data.

[0126] In the embodiment of the present application, the target network performance data is evaluated according to the target evaluation strategy, and the performance evaluation result corresponding to the to-be-tested device is obtained, as shown in FIG. 8, including the following steps B1-B3: Figure 4

[0127] Step B1, if the performance evaluation type is a real-time network service performance evaluation type, third index data corresponding to the preset index is obtained from the first network service performance data, and fourth index data corresponding to the preset index is obtained from the second network service data.

[0128] ​Specifically, when determining that the performance evaluation type is the real-time network service performance evaluation type, first, the relevant data obtained is determined. In this case, the first network service performance data and the second network service performance data have been obtained through corresponding operations. The first network service performance data is collected from the device under test and contains a series of index information such as ping, upload, download, video playback, and other indicators that can reflect the actual running status of the device under test in terms of network service. The second network service performance data is obtained from the reference machine and also covers various indicators for measuring network service performance.

[0129] Next, for the pre-set key indicators for evaluation, that is, the pre-set indicators (these pre-set indicators are usually determined according to specific evaluation requirements and the aspects of service performance that are focused on, such as indicators that may focus on ping round-trip time, upload speed, download speed, video playback smoothness, etc.), the specific data corresponding to these pre-set indicators is accurately extracted from the first network service performance data and defined as third index data. Similarly, the corresponding fourth index data is accurately obtained from the second network service performance data according to the requirements of the pre-set indicators. Through such operations, the corresponding specific index data is extracted from the two sets of network service performance data according to the pre-set indicators, and the subsequent comparative analysis is prepared.

[0130] Step B2, compare the third index data and the fourth index data associated with each pre-set indicator.

[0131] Specifically, after obtaining the third index data and the fourth index data corresponding to each pre-set indicator from the first network service performance data and the second network service performance data, the comparison operation begins.

[0132] For each pre-set indicator, the third index data obtained from the device under test is compared with the fourth index data obtained from the reference machine. For example, if the pre-set indicator is the ping round-trip time, then the third index data about the ping round-trip time extracted from the first network service performance data is compared with the fourth index data about the ping round-trip time extracted from the second network service performance data to view the size relationship between the two values. The same is true for other pre-set indicators such as upload speed, video playback smoothness, etc. Each pre-set indicator is sequentially compared and analyzed in detail to determine the performance difference between the device under test and the reference machine in each key indicator.

[0133] Step B3, if at least one third index data associated with the preset index is lower than the fourth index data, determining that the performance evaluation result is that the service performance of the device under test is lower than the service performance of the reference device; or, if all third index data associated with the preset index is higher than the fourth index data, determining that the performance evaluation result is that the service performance of the device under test is higher than the service performance of the reference device.

[0134] Specifically, if it is found that at least one third index data associated with the preset index is lower than the corresponding fourth index data among all the compared preset indexes, it means that the device under test performs worse than the reference device on these key indexes, and thus it can be determined that the performance evaluation result is that the service performance of the device under test is lower than the service performance of the reference device.

[0135] On the contrary, if it is found that all third index data associated with the preset index is higher than the corresponding fourth index data after comparison, it means that the device under test performs better than the reference device on each key index, and thus it can be determined that the performance evaluation result is that the service performance of the device under test is higher than the service performance of the reference device. In this way, the advantages and disadvantages of the device under test and the reference device in real-time network service performance can be accurately determined according to the comparison of the preset index data.

[0136] In the embodiments of the present application, the target network performance data is evaluated according to the target evaluation strategy, and the performance evaluation result corresponding to the device under test is obtained, as shown in FIG. 6, including the following steps C1-C3. Figure 5

[0137] Step C1, if the performance evaluation type is a network performance comprehensive evaluation type, obtaining the current performance of the device under test by comparing the first network performance data and the second network performance data.

[0138] Specifically, when the performance evaluation type is a network performance comprehensive evaluation type, the control unit first acquires the first network performance data (including the network basic performance data of the device under test such as RSSI, RSRP, SNR, RSRQ, etc. and network service performance data such as ping, upload, download, video playback, etc.) and the second network performance data (the corresponding network performance data of the reference device) based on a specific connection mode. These data are all time-stamped. Then the data are processed according to the set rules, and each index in the first network performance data and the second network performance data is directly compared, for example, the RSSI value and the ping value of the device under test and the reference device are compared. Through the comparison and analysis of these indexes, the network performance status of the device under test relative to the reference device at the current time is determined, that is, the current performance of the device under test is obtained.

[0139] ​Step C2, compare the first historical network performance data and the second historical network performance data to obtain the historical performance of the device under test.

[0140] Specifically, in the case of the performance evaluation type being the network performance comprehensive evaluation type, the control unit will synchronize the first historical network performance data (historical network performance data of the device under test, covering network basic performance and network service performance related indicators at different time points in the past) and the second historical network performance data (historical network performance data of the reference machine) from the specified location or the storage unit permanent area as needed. Then, the corresponding indicators in the first historical network performance data and the second historical network performance data are compared and analyzed, such as checking the download speed change of the device under test and the reference machine in a certain period of time in the past, so as to obtain the historical performance of the device under test.

[0141] Step C3, compare the first network performance data and the first historical network performance data to obtain the change trend of the network performance of the device under test over time.

[0142] Specifically, there are many ways to analyze the change trend, such as drawing a line chart, taking time as the horizontal axis and the selected network performance indicator value (such as bandwidth value) as the vertical axis, marking the current and each historical time point data on the chart, and observing the trend of the line to intuitively judge whether it is rising, falling or stable. The trend can also be quantitatively analyzed by calculating the change rate of the performance indicator value between adjacent time points, such as calculating the growth rate or decline rate of the bandwidth in each time period. Such trend analysis operations are performed on each network performance indicator, so as to fully grasp the change trend of the network performance of the device under test over time. For example, it is found that the network bandwidth has shown a gradual downward trend in the past few months, while the packet loss rate has shown an upward trend, which indicates that the network performance is gradually deteriorating.

[0143] Step C4, analyze the current performance and the historical performance to obtain the performance difference of the device under test.

[0144] Specifically, for each network performance indicator, the current performance situation is compared with the corresponding data in the historical performance situation. For example, the difference between the current network bandwidth and the historical average bandwidth is observed, or the fluctuation range change of the current delay time compared with the historical same period is compared. Not only the difference in the numerical value is focused on, but also the correlation between the difference at different times and different performance indicators is analyzed. For example, it is found that not only the bandwidth is reduced, but also the delay is increased, and this situation is more serious than most of the time in history, which means that the measured device has a more significant difference in network performance, and there may be potential problems affecting the network operation. Through the comprehensive comparison and analysis of various indicators, the complete performance difference of the measured device is sorted out, and it is clear which performance indicators have changed significantly and the degree of change.

[0145] Step C5, generating a performance evaluation result based on the performance difference of the measured device and the change trend.

[0146] Specifically, according to the performance difference of the measured device, the specific advantages and disadvantages of the current network performance compared with the historical performance in each indicator are determined, and according to the change trend of the network performance over time, it is judged whether the change is a short-term fluctuation or a long-term stable trend, and the coordination between different performance indicators is changed. For example, if the performance difference shows that the current bandwidth has decreased significantly and the change trend shows that this decrease has lasted for a long time, the bandwidth related influencing factors need to be focused on.

[0147] According to the above comprehensive analysis, a performance evaluation result is generated. The evaluation result content includes the overall evaluation of the network performance of the measured device (such as good, general, poor, etc.), the specific network performance indicators with problems and the abnormal situations they show (such as insufficient bandwidth, high delay, serious packet loss, etc.), and some preliminary suggestions can also be given based on the analysis, such as whether the device needs to be checked and repaired, whether the network configuration needs to be optimized and adjusted, etc.

[0148] In the embodiments of the present application, the target network performance data is evaluated according to the target evaluation strategy, and the performance evaluation result corresponding to the measured device is obtained, as shown in Figure 6 The following steps D1-D4 are included:

[0149] Step D1, if the performance evaluation type is a batch network performance evaluation type, the network basic performance data of the measured device in the current batch is processed according to the preset processing rule, and the processed network basic performance data of the measured device in the current batch is obtained.

[0150] Specifically, when the performance evaluation type is the batch network performance evaluation type, the control unit first acquires network basic performance data (including indicators such as RSSI, RSRP, SNR, RSRQ, etc.) of all the measured devices in the current batch based on the connection mode of remote control of the mass of measured devices through the wireless network. Then, the acquired network basic performance data is processed according to a preset processing rule. For example, according to the rule of moving average in a time period of 10 seconds, or the rule of moving average in a position distance difference of 100 m, etc., the RSSI, RSRP, and other network basic performance data of each measured device are processed respectively, and through the processing process, the processed network basic performance data of the measured devices in the current batch is finally obtained, so as to provide more accurate and targeted data basis for subsequent evaluation and analysis.

[0151] Step D2, acquiring the space-time state information corresponding to the network basic performance data of the measured devices in the current batch.

[0152] Specifically, the space-time state information corresponding to the network basic performance data of the measured devices in the current batch is acquired. The control unit acquires the time and position information data corresponding to the network basic performance data based on the same wireless network remote control connection mode at the same time of acquiring the network basic performance data, and these time and position information data constitute the space-time state information corresponding to the network basic performance data. It clearly shows the network basic performance data of each measured device at a specific time in a specific space position, and provides key space-time dimension information for further correlation analysis.

[0153] Step D3, acquiring the corresponding correlation network basic performance data of each measured device contained in the current batch by using the space-time state information.

[0154] Specifically, the corresponding correlation network basic performance data of each measured device contained in the current batch is acquired by using the space-time state information, including: acquiring a network map data model, wherein the network map data model includes historical network performance data corresponding to different historical positions of different measured devices and different historical times; acquiring a historical network performance data set corresponding to the measured devices contained in the current batch from the network map data model; and filtering the corresponding correlation network basic performance data of each measured device in the current batch from the historical network performance data set by using the space-time state information.

[0155] First, obtain the network map data model. This model stores historical network performance data of multiple devices under test at different historical locations and different historical times. This data can be obtained by continuously collecting and organizing network basic performance data (such as RSSI / RSRP / SNR / RSRQ, etc.) and network service performance data (such as ping, upload, download, video playback, etc.) of each device under test under different space-time conditions, and is organized according to geographical map information and time.

[0156] Next, from this network map data model, according to the requirements of the current batch, obtain the historical network performance data set corresponding to the devices under test contained in the current batch. This step can extract the historical network performance data set related to the current batch from the model by querying and filtering according to the identification or other specific conditions of the device under test.

[0157] Finally, use the space-time state information of each device under test in the current batch to filter from the historical network performance data set. Specifically, according to the time information and spatial location information, match with the data in the historical network performance data set to find the network basic performance data associated with each device under test at a specific time and location, so as to analyze and process these data for subsequent network performance evaluation and optimization of the current batch.

[0158] Step D4, compare the network basic performance data of each device under test contained in the current batch with the associated network basic performance data to obtain the performance evaluation result corresponding to the device under test.

[0159] Specifically, after obtaining the processed network basic performance data of the device under test in the current batch and the corresponding associated network basic performance data, the comparison operation is started. The processed network basic performance data and the associated network basic performance data are compared one by one according to each index, such as the values of RSSI, RSRP, etc. If it is found that the processed network basic performance data is lower than the associated network basic performance data in some indicators, it means that the network basic performance of the device under test is relatively poor under the corresponding space-time condition; on the contrary, if the processed network basic performance data is higher than the associated network basic performance data in each indicator, it means that the network basic performance of the device under test is good under the space-time condition. Through such comprehensive comparison and analysis, the performance evaluation result corresponding to the device under test is finally obtained, so that the network basic performance of the device under test in the current batch can be clearly judged, so that subsequent corresponding measures can be taken, such as optimizing and adjusting the devices with poor performance.

[0160] In the embodiments of the present application, the performance evaluation result is used to perform corresponding processing operations on the device operation log, including the following cases:

[0161] ①When the comparison and evaluation of the network basic performance data of the measured device and the reference device is completed, if the measured device is determined to be inferior to the reference device, the corresponding action will be triggered. The network, system, application, etc. logs of the measured device in the current period are saved in the permanent area of the storage unit or uploaded to the designated location for further detailed analysis of the specific running status and possible problems of the device during the poor performance. Conversely, if the measured device is determined to be not inferior to the reference device, the network, system, application logs in the temporary area of the storage unit are deleted according to the set deletion rule, thereby saving the storage unit space and avoiding unnecessary log accumulation.

[0162] ②After initiating the service instruction to the measured device and the reference device to obtain the related data, and after the data processing and comparison of the network service performance data of the two devices, if the network service performance of the measured device is determined to be worse than that of the reference device, an alarm action will be triggered, and the network, system, application, etc. logs of the measured device in the current period are saved in the permanent area of the storage unit or uploaded to the designated location for further in-depth investigation of the possible reasons for the poor service performance. If the network service performance of the measured device is not worse than that of the reference device, the network, system, application logs in the temporary area of the storage unit are deleted according to the set deletion rule, thereby saving the storage unit space and ensuring the effective use of storage resources.

[0163] ③After completing the real-time and historical network performance data comparison, the device running logs are processed according to the evaluation results. If it is found that the network performance of the measured device at the current or historical specific time position has obvious problems, such as being worse than the reference device or showing a significant downward trend in its own historical data, etc., the network, system, application, etc. logs of the measured device in the corresponding period are saved in the permanent area of the storage unit or uploaded to the designated location for detailed analysis of these problem periods. Conversely, if the evaluation result shows that the network performance of the device is stable and has no obvious abnormalities, the logs in the temporary area of the storage unit are deleted according to the set deletion rule, thereby saving the storage unit space.

[0164] ④After comparing and evaluating the current network basic performance data of the measured device with its own historical record information, if it is determined that the performance of the device has changed, such as performance decline, etc., the network, system, application, etc. logs of the measured device in the current period are saved in the permanent area of the storage unit or uploaded to the designated location for in-depth analysis of the reasons for the performance change. If the comparison result shows that the performance of the device is stable and has no obvious abnormalities, the network, system, application logs in the temporary area of the storage unit are deleted according to the set deletion rule, thereby saving the storage unit space and maintaining the efficiency of log management.

[0165] ⑤ After conducting a network fundamental performance evaluation on a large number of tested devices, and after data processing and comparison with network fundamental performance data at relevant time points, alarm actions will be triggered for devices judged to have poor performance. Simultaneously, the network, system, and application logs of these devices for the current time period will be saved in a specific area of ​​their internal storage space, facilitating subsequent individual analysis and troubleshooting of these poorly performing devices. For devices with normal performance, network, system, and application logs in temporary areas of their internal storage space will be deleted cyclically according to set deletion rules to conserve internal storage space and ensure that storage resources can meet the storage needs during large-scale testing.

[0166] The method provided in this application obtains network performance evaluation instructions for intelligent connected vehicles, clarifying the performance evaluation type and target evaluation strategy. This allows testing to move beyond the previous single, isolated road test data model, enabling data acquisition from multiple perspectives based on different evaluation types. Secondly, the response instructions utilize the performance evaluation type to obtain target network performance data associated with the device under test, as well as device operation logs, comprehensively covering various situations recorded during device operation. This achieves comprehensive quantitative recording of network-related indicators, changing the previous situation of lacking comprehensive quantitative records. Furthermore, the process of evaluating the target network performance data according to the target evaluation strategy to obtain performance evaluation results involves comparative analysis of data under different conditions, compensating for the previous lack of horizontal and vertical data analysis.

[0167] This embodiment also provides a performance evaluation device for intelligent connected vehicles, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0168] This embodiment provides a performance evaluation device for intelligent connected vehicles, such as... Figure 7 As shown, it includes:

[0169] The acquisition module 801 is used to acquire network performance evaluation instructions for intelligent connected vehicles, wherein the network performance evaluation instructions include performance evaluation type and target evaluation strategy.

[0170] The response module 802 is used to respond to network performance evaluation commands, obtain target network performance data associated with the device under test in the intelligent connected vehicle using the performance evaluation type, and obtain the device operation log corresponding to the device under test.

[0171] The evaluation module 803 is configured to evaluate the target network performance data according to a target evaluation strategy, to obtain a performance evaluation result corresponding to the device under test, and to perform a corresponding processing operation on the device running log according to the performance evaluation result.

[0172] In the embodiment of the present application, the response module 802 is configured to, if the performance evaluation type is a real-time network basic coverage performance evaluation type, acquire first network basic performance data of the device under test and second network basic performance data of the reference machine, perform data smoothing processing on the first network basic performance data and the second network basic performance data according to a preset processing rule, and use the processed first network basic performance data and the processed second network basic performance data as the target network performance data; if the performance evaluation type is a real-time network service performance evaluation type, acquire first network service performance data of the device under test and second network service performance data of the reference machine, perform data smoothing processing on the first network service performance data and the second network service performance data according to a preset processing rule, and use the processed first network service performance data and the processed second network service performance data as the target network performance data; if the performance evaluation type is a network performance comprehensive evaluation type, acquire first network performance data of the device under test and second network performance data of the reference machine, determine first space-time information corresponding to the first network performance data and second space-time information corresponding to the second network performance data, acquire first historical network performance data associated with the first space-time information and second historical network performance data matched with the second space-time information, and use the first network performance data, the second network performance data, the first historical network performance data and the second historical network performance data as the target network performance data; if the performance evaluation type is a batch network performance evaluation type, acquire network basic performance data of the device under test included in a current batch, and use the network basic performance data of the device under test included in the current batch as the target network performance data.

[0173] In the embodiment of the present application, the response module 802 is configured to acquire a network map data model, wherein the network map data model includes historical network performance data corresponding to different historical positions and different historical times of the device under test; extract time information and spatial position information from the first space-time information; acquire a historical relative position matched with the spatial position information from the network map data model, and acquire historical network performance data corresponding to different historical times associated with the historical relative position; acquire first historical network performance data matched with the time information from the historical network performance data corresponding to different historical times associated with the first historical relative position, and acquire second historical network performance data matched with the second space-time information from a historical network performance data set corresponding to the reference machine.

[0174] In the embodiments of the present application, the response module 802 is configured to acquire potential historical position information adjacent to the spatial position information from the network map data model; calculate a position difference based on the spatial position information and the potential historical position information, and take the potential historical position information with a position difference less than a preset distance difference value as candidate historical position information; calculate a first speed value based on a distance change between the spatial position information and the candidate historical position information and a first time difference, wherein the first time difference is calculated based on the time information and historical time information corresponding to the candidate historical position information; calculate a second speed based on a distance change between the candidate historical position information at different historical time and a second time difference, wherein the second time difference is calculated based on the candidate historical position information corresponding to different historical time information; calculate a speed difference value between the first speed value and the second speed value, and determine a historical relative position by using the candidate historical position information with a speed difference value less than a preset speed difference value.

[0175] In the embodiments of the present application, the device further comprises an updating module configured to perform correlation analysis on the first network performance data and historical network performance data in the network map data model to obtain a plurality of correlation data between the first network performance data and the historical network performance data and a spatio-temporal feature; update the correlation data to the network map data model according to the spatio-temporal feature; and fuse the plurality of correlation data between the first network performance data and the historical network performance data according to a preset algorithm in the network map data model to obtain an optimized network map data model.

[0176] In the embodiments of the present application, the response module 802 is configured to obtain a first historical network data set and a second historical network data set, wherein the first historical network data set comprises historical network performance data corresponding to different historical times, and the second historical network data set comprises historical network performance data corresponding to different locations; extract a first time and a first spatial position from the first spatio-temporal information, and extract a second time and a second spatial position from the second spatio-temporal information; obtain a first historical time matched with the time characteristic of the first time, and a second historical time matched with the time characteristic of the second time, and obtain first candidate network performance data corresponding to the first historical time and second candidate network performance data corresponding to the second historical time from the first original data set; obtain a plurality of historical relative positions associated with the first spatial position and a plurality of historical relative positions associated with the second spatial position, wherein the plurality of historical relative positions associated with the first spatial position are filtered according to the position difference between the first spatial position, and the plurality of historical relative positions associated with the second spatial position are filtered according to the position difference between the second spatial position; obtain a first historical relative position from the plurality of historical relative positions associated with the first spatial position by using the first speed difference, and obtain a second historical relative position from the plurality of historical relative positions associated with the second spatial position by using the second speed difference; obtain third candidate network performance data corresponding to the first historical relative position and fourth candidate network performance data corresponding to the second historical relative position from the second original data set; construct first historical network performance data associated with the first spatio-temporal information based on the first candidate network performance data and the third candidate network performance data, and construct second historical network performance data associated with the second spatio-temporal information based on the second candidate network performance data and the fourth candidate network performance data.

[0177] In the embodiments of the present application, the evaluation module 803 is configured to, if the performance evaluation type is a real-time network basic coverage performance evaluation type, obtain first index data corresponding to a preset index from the first network basic performance data, and obtain second index data corresponding to the preset index from the second network basic data; compare the first index data and the second index data associated with each preset index; if the first index data associated with at least one preset index is lower than the second index data, determine that the performance evaluation result is that the basic performance of the to-be-tested device is lower than the basic performance of the reference machine; or, if the first index data associated with the preset index is all higher than the second index data, determine that the performance evaluation result is that the basic performance of the to-be-tested device is higher than the basic performance of the reference machine.

[0178] In the embodiment of the present application, the evaluation module 803 is configured to, if the performance evaluation type is the real-time network service performance evaluation type, acquire third index data corresponding to the preset index from the first network service performance data and acquire fourth index data corresponding to the preset index from the second network service performance data; compare the third index data and the fourth index data associated with each preset index; if the third index data associated with at least one preset index is lower than the fourth index data, determine that the performance evaluation result is that the service performance of the device under test is lower than the service performance of the reference machine; or, if the third index data associated with each preset index is higher than the fourth index data, determine that the performance evaluation result is that the service performance of the device under test is higher than the service performance of the reference machine.

[0179] In the embodiment of the present application, the evaluation module 803 is configured to, if the performance evaluation type is the network performance comprehensive evaluation type, obtain the current performance situation of the device under test by comparing the first network performance data and the second network performance data; obtain the historical performance situation of the device under test by comparing the first historical network performance data and the second historical network performance data; obtain the change trend of the network performance of the device under test with time by comparing the first network performance data and the first historical network performance data; obtain the performance difference situation of the device under test by analyzing the current performance situation and the historical performance situation; and generate the performance evaluation result based on the performance difference situation and the change trend of the device under test.

[0180] In the embodiment of the present application, the evaluation module 803 is configured to, if the performance evaluation type is the batch network performance evaluation type, process the network basic performance data of the device under test in the current batch according to a preset processing rule to obtain the processed network basic performance data of the device under test in the current batch; acquire the space-time state information corresponding to the network basic performance data of the device under test in the current batch.

[0181] Obtain the corresponding associated network basic performance data of each device under test in the current batch by using the space-time state information; and compare the network basic performance data of each device under test in the current batch with the associated network basic performance data to obtain the performance evaluation result corresponding to the device under test.

[0182] In the embodiment of the present application, the evaluation module 803 is configured to acquire a network map data model, wherein the network map data model includes historical network performance data corresponding to different historical positions of the device under test and different historical times of different devices under test; acquire a historical network performance data set corresponding to the device under test included in the current batch from the network map data model; and filter the corresponding associated network basic performance data of each device under test in the current batch from the historical network performance data set by using the space-time state information.

[0183] Please refer to Figure 8 , Figure 8This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 8 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 8 Take a processor 10 as an example.

[0184] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0185] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[0186] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device as shown by a landing page for an app. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, which can be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0187] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0188] The computer device also comprises a communication interface 30 for the computer device to communicate with other devices or communication networks.

[0189] The embodiments of the present application also provide a computer readable storage medium, the method according to the embodiments of the present application can be implemented in hardware, firmware, or be implemented as computer code recorded in a storage medium, or be implemented through network downloading and originally stored in a remote storage medium or a non-transitory machine readable storage medium and to be stored in a local storage medium, so that the method described herein can be processed by such software on a storage medium using a general computer, a special processor or programmable or special hardware. Wherein, the storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk or a solid state disk, etc. Further, the storage medium can also include a combination of the above-mentioned types of memories. It can be understood that the computer, the processor, the microprocessor controller or the programmable hardware includes a storage component that can store or receive software or computer code, when the software or computer code is accessed and executed by the computer, the processor or the hardware, the method shown in the above embodiments is implemented.

[0190] Although the embodiments of the present application are described in conjunction with the accompanying drawings, various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present application, and such modifications and changes fall within the scope defined by the appended claims.

Claims

1. A method for performance evaluation of an intelligent connected vehicle, characterized in that, The method comprises: obtaining a network performance evaluation instruction for an intelligent connected vehicle, wherein the network performance evaluation instruction comprises a performance evaluation type and a target evaluation strategy; in response to the network performance evaluation instruction, obtaining target network performance data associated with a measured device in the intelligent connected vehicle using the performance evaluation type, and obtaining a device operation log corresponding to the measured device; evaluating the target network performance data according to the target evaluation strategy to obtain a performance evaluation result corresponding to the measured device, and performing a corresponding processing operation on the device operation log according to the performance evaluation result; the target network performance data associated with the measured device in the intelligent connected vehicle is obtained using the performance evaluation type, comprising: if the performance evaluation type is a real-time network basic coverage performance evaluation type, first network basic performance data of the measured device and second network basic performance data of a reference machine are obtained, the first network basic performance data and the second network basic performance data are subjected to data smoothing processing according to a preset processing rule, and the processed first network basic performance data and the processed second network basic performance data are taken as the target network performance data; wherein the first network basic performance data and the second network basic performance data comprise RSSI, RSRP, SNR and RSRQ; if the performance evaluation type is a real-time network service performance evaluation type, first network service performance data of the measured device and second network service performance data of a reference machine are obtained, the first network service performance data and the second network service performance data are subjected to data smoothing processing according to a preset processing rule, and the processed first network service performance data and the processed second network service performance data are taken as the target network performance data; wherein the first network service performance data and the second network service performance data comprise data corresponding to ping, upload, download and video playback; if the performance evaluation type is a network performance comprehensive evaluation type, first network performance data of the measured device and second network performance data of a reference machine are obtained, first space-time information corresponding to the first network performance data and second space-time information corresponding to the second network performance data are determined, first historical network performance data associated with the first space-time information and second historical network performance data matching the second space-time information are obtained, and the first network performance data, the second network performance data, the first historical network performance data and the second historical network performance data are taken as the target network performance data; if the performance evaluation type is a batch network performance evaluation type, network basic performance data of the measured device contained in the current batch is obtained, and the network basic performance data of the measured device contained in the current batch is taken as the target network performance data; the target network performance data is evaluated according to the target evaluation strategy to obtain the performance evaluation result corresponding to the measured device, comprising: If the performance evaluation type is a real-time network infrastructure coverage performance evaluation type, first index data corresponding to a preset index is obtained from the processed first network infrastructure performance data, and second index data corresponding to the preset index is obtained from the processed second network infrastructure performance data; the first index data and the second index data associated with each preset index are compared; if the first index data associated with at least one preset index is lower than the second index data, it is determined that the performance evaluation result is that the infrastructure performance of the device under test is lower than the infrastructure performance of the reference machine; or, if the first index data associated with each preset index is higher than the second index data, it is determined that the performance evaluation result is that the infrastructure performance of the device under test is higher than the infrastructure performance of the reference machine; The performance evaluation result corresponding to the device under test is obtained by evaluating the target network performance data according to the target evaluation strategy, including: if the performance evaluation type is a real-time network service performance evaluation type, third index data corresponding to a preset index is obtained from the processed first network service performance data, and fourth index data corresponding to the preset index is obtained from the processed second network service performance data; the third index data and the fourth index data associated with each preset index are compared; if the third index data associated with at least one preset index is lower than the fourth index data, it is determined that the performance evaluation result is that the service performance of the device under test is lower than the service performance of the reference machine; or, if the third index data associated with each preset index is higher than the fourth index data, it is determined that the performance evaluation result is that the service performance of the device under test is higher than the service performance of the reference machine; The performance evaluation result corresponding to the device under test is obtained by evaluating the target network performance data according to the target evaluation strategy, including: if the performance evaluation type is a network performance comprehensive evaluation type, the current performance of the device under test is obtained by comparing the first network performance data and the second network performance data; the historical performance of the device under test is obtained by comparing the first historical network performance data and the second historical network performance data; the change trend of the network performance of the device under test over time is obtained by comparing the first network performance data and the first historical network performance data; the performance difference of the device under test is obtained by analyzing the current performance and the historical performance; and the performance evaluation result is generated based on the performance difference of the device under test and the change trend. The evaluating the target network performance data according to the target evaluation strategy to obtain the performance evaluation result corresponding to the DUT comprises: if the performance evaluation type is a batch network performance evaluation type, processing network basic performance data of the DUT in a current batch according to a preset processing rule to obtain processed network basic performance data of the DUT in the current batch; obtaining space-time state information corresponding to the network basic performance data of the DUT in the current batch; obtaining associated network basic performance data of each DUT included in the current batch by using the space-time state information; comparing the network basic performance data of each DUT included in the current batch with the associated network basic performance data to obtain the performance evaluation result corresponding to the DUT.

2. The method of claim 1, wherein, The obtaining the first historical network performance data associated with the first space-time information and the second historical network performance data matched with the second space-time information comprises: obtaining a network map data model, wherein the network map data model comprises historical network performance data corresponding to different historical positions and different historical times of the DUT; extracting time information and spatial position information from the first space-time information; obtaining a historical relative position matched with the spatial position information from the network map data model, and obtaining historical network performance data corresponding to different historical times associated with the historical relative position; obtaining the first historical network performance data matched with the time information from the historical network performance data corresponding to different historical times associated with the first historical relative position, and obtaining the second historical network performance data matched with the second space-time information from the historical network performance data set corresponding to the reference machine.

3. The method of claim 2, wherein, The obtaining the historical relative position matched with the spatial position information from the network map data model comprises: obtaining potential historical position information adjacent to the spatial position information from the network map data model; calculating a position difference based on the spatial position information and the potential historical position information, and taking potential historical position information with a position difference less than a preset distance difference value as candidate historical position information; calculating a first speed value according to a distance change between the spatial position information and the candidate historical position information and a first time difference, wherein the first time difference is calculated based on the time information and historical time information corresponding to the candidate historical position information; calculating a second speed value according to a distance change between different historical times of the candidate historical position information and a second time difference, wherein the second time difference is calculated based on different historical time information corresponding to the candidate historical position information; calculating a speed difference value between the first speed value and the second speed value, and determining the historical relative position by using candidate historical position information with a speed difference value less than a preset speed difference value.

4. The method of claim 2, wherein, The method further comprises: Correlation analysis is performed on the first network performance data and historical network performance data in the network map data model to obtain spatiotemporal characteristics and a plurality of correlation data between the first network performance data and the historical network performance data; The correlation data is updated to the network map data model according to the spatiotemporal characteristics; A plurality of correlation data between the first network performance data and the historical network performance data are fused according to a preset algorithm in the network map data model to obtain an optimized network map data model.

5. The method of claim 1, wherein, The associated network basic performance data corresponding to each device under test in the current batch is obtained by using the spatiotemporal state information, including: Obtaining a network map data model, wherein the network map data model includes historical network performance data corresponding to a plurality of devices under test at different historical positions of the devices under test and at different historical times; Obtaining a historical network performance data set corresponding to the devices under test in the current batch from the network map data model; Using the spatiotemporal state information to filter the associated network basic performance data corresponding to each device under test in the current batch from the historical network performance data set.

6. A performance evaluation device of an intelligent connected vehicle, characterized in that, The device includes: An acquisition module is configured to acquire a network performance evaluation instruction for an intelligent connected vehicle, wherein the network performance evaluation instruction includes a performance evaluation type and a target evaluation strategy; A response module is configured to respond to the network performance evaluation instruction, acquire target network performance data associated with a device under test in the intelligent connected vehicle by using the performance evaluation type, and acquire a device operation log corresponding to the device under test; An evaluation module is configured to evaluate the target network performance data according to the target evaluation strategy to obtain a performance evaluation result corresponding to the device under test, and perform a corresponding processing operation on the device operation log according to the performance evaluation result; The response module is configured to, if the performance evaluation type is a real-time network basic coverage performance evaluation type, acquire first network basic performance data of the device under test and second network basic performance data of a reference device, perform data smoothing processing on the first network basic performance data and the second network basic performance data according to a preset processing rule, and use the processed first network basic performance data and the processed second network basic performance data as the target network performance data; wherein the first network basic performance data and the second network basic performance data include RSSI, RSRP, SNR, and RSRQ. The response module is configured to, if the performance evaluation type is a real-time network service performance evaluation type, acquire first network service performance data of the device under test and second network service performance data of the reference machine, perform data smoothing processing on the first network service performance data and the second network service performance data according to a preset processing rule, and use the processed first network service performance data and the processed second network service performance data as the target network performance data; wherein the first network service performance data and the second network service performance data include data corresponding to ping, upload, download, and video playback. The response module is configured to, if the performance evaluation type is a network performance comprehensive evaluation type, acquire first network performance data of the device under test and second network performance data of the reference machine, determine first space-time information corresponding to the first network performance data and second space-time information corresponding to the second network performance data, acquire first historical network performance data associated with the first space-time information and second historical network performance data matched with the second space-time information, and use the first network performance data, the second network performance data, the first historical network performance data, and the second historical network performance data as the target network performance data. The response module is configured to, if the performance evaluation type is a batch network performance evaluation type, acquire network basic performance data of the device under test included in a current batch, and use the network basic performance data of the device under test included in the current batch as the target network performance data. The evaluation module is configured to, if the performance evaluation type is a real-time network basic coverage performance evaluation type, acquire first index data corresponding to a preset index from the processed first network basic performance data, and acquire second index data corresponding to the preset index from the processed second network basic performance data; compare the first index data and the second index data associated with each preset index; if the first index data associated with at least one preset index is lower than the second index data, determine that the performance evaluation result is that the basic performance of the device under test is lower than the basic performance of the reference machine; or, if the first index data associated with the preset index is all higher than the second index data, determine that the performance evaluation result is that the basic performance of the device under test is higher than the basic performance of the reference machine. The evaluation module is configured to, if the performance evaluation type is a real-time network service performance evaluation type, acquire third index data corresponding to a preset index from the processed first network service performance data, and acquire fourth index data corresponding to the preset index from the processed second network service performance data; compare the third index data and the fourth index data associated with each preset index; if the third index data associated with at least one preset index is lower than the fourth index data, determine that the performance evaluation result is that the service performance of the device under test is lower than the service performance of the reference machine; or, if the third index data associated with each preset index is higher than the fourth index data, determine that the performance evaluation result is that the service performance of the device under test is higher than the service performance of the reference machine. The evaluation module is configured to, if the performance evaluation type is a network performance comprehensive evaluation type, obtain a current performance condition of the device under test by comparing the first network performance data and the second network performance data; obtain a historical performance condition of the device under test by comparing the first historical network performance data and the second historical network performance data; obtain a change trend of the network performance of the device under test over time by comparing the first network performance data and the first historical network performance data; obtain a performance difference condition of the device under test by analyzing the current performance condition and the historical performance condition; and generate the performance evaluation result based on the performance difference condition of the device under test and the change trend. The evaluation module is configured to, if the performance evaluation type is a batch network performance evaluation type, process network basic performance data of the device under test in a current batch according to a preset processing rule to obtain processed network basic performance data of the device under test in the current batch; acquire space-time state information corresponding to the network basic performance data of the device under test in the current batch; acquire corresponding associated network basic performance data of each device under test included in the current batch by using the space-time state information; and compare the network basic performance data of each device under test included in the current batch with the associated network basic performance data to obtain a performance evaluation result corresponding to the device under test.

7. An electronic device, comprising: The computer readable storage medium has stored thereon computer instructions for causing a computer to execute the method of any one of claims 1 to 5. The computer readable storage medium has stored thereon computer instructions for causing a computer to execute the method of any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, ​

Images