Performance evaluation method, device and equipment of intelligent network connection vehicle and medium

Abstract

The invention relates to the field of vehicle communication, in particular to a performance evaluation method, device and equipment of an intelligent network connection vehicle and a medium. By acquiring the network performance evaluation instruction for the intelligent network connection vehicle, the performance evaluation type and the target evaluation strategy are clarified, so that the test is not limited to a previous single isolated road test data mode any more, and data can be acquired from multiple aspects according to different evaluation types. And secondly, in response to the instruction, the target network performance data associated with the tested equipment and the equipment operation log are obtained by using the performance evaluation type, and various condition records in the equipment operation process are comprehensively covered, so that comprehensive quantitative recording of network related indexes is realized, and the condition of lack of comprehensive quantitative recording in the past is changed. Moreover, in the process of evaluating the target network performance data according to the target evaluation strategy to obtain the performance evaluation result, data comparative analysis under different conditions is involved, and the defect of lack of transverse and longitudinal data analysis in the prior art is overcome.

Description

Technical Field

[0001] The present invention relates to the field of vehicle communications, and in particular to a performance evaluation method, device, equipment and medium for an intelligent networked vehicle. Background Art

[0002] As a basic technology for intelligent connected vehicles, the importance of network communication systems cannot be underestimated. In the field of cockpit navigation of intelligent connected vehicles, it ensures the real-time transmission of map data and the accurate issuance of navigation instructions, allowing drivers to accurately obtain route guidance. In terms of autonomous driving, network communication systems are even more critical. The vehicle's sensor data needs to be transmitted and interacted quickly and accurately through it, so that the autonomous driving system can make timely decisions based on this information to ensure safe and smooth driving.

[0003] But for now, there are many deficiencies in the testing and analysis of intelligent connected vehicles. Most of the work is only analyzed around isolated road test data. R&D personnel often do not make comprehensive quantitative records of various network-related indicators, which leads to many problems. For example, due to the lack of horizontal comparative analysis, R&D personnel cannot clearly know the differences in network performance between different vehicles or the same vehicle in different environments. The lack of longitudinal historical data analysis makes it difficult for them to grasp the trend of network performance over time. What's more serious is that there is no automated testing and analysis method in the entire industry that combines large-scale batch testing with historical big data, which greatly limits the optimization and improvement of the network communication system of intelligent connected vehicles and is not conducive to its further development. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a performance evaluation method, device, equipment and medium for an intelligent connected vehicle to solve the problems that existing test analysis is mostly limited to isolated road test data, lacks comprehensive quantitative records of network-related indicators and horizontal and vertical data analysis, and lacks large-scale batch testing combined with historical big data for automated test analysis.

[0005] In a first aspect, an embodiment of the present invention provides a performance evaluation method for an intelligent connected vehicle, the method comprising:

[0006] Obtaining a network performance evaluation instruction for an intelligent connected vehicle, wherein the network performance evaluation instruction includes a performance evaluation type and a target evaluation strategy;

[0007] In response to the network performance evaluation instruction, using the performance evaluation type to obtain target network performance data associated with the device under test in the intelligent connected vehicle, and obtaining a device operation log corresponding to the device under test;

[0008] The target network performance data is evaluated according to the target evaluation strategy to obtain a performance evaluation result corresponding to the device under test, and corresponding processing operations are performed on the device operation log according to the performance evaluation result.

[0009] Furthermore, the step of obtaining target network performance data associated with the device under test in the intelligent connected vehicle by using the performance evaluation type includes:

[0010] If the performance evaluation type is a real-time network basic coverage performance evaluation type, the first network basic performance data of the device under test and the second network basic performance data of the reference gold machine are obtained, the first network basic performance data and the second network basic performance data are smoothed according to a preset processing rule, and the processed first network basic performance data and the processed second network basic performance data are used as the target network performance data;

[0011] If the performance evaluation type is a real-time network service performance evaluation type, the first network service performance data of the device under test and the second network service performance data of the reference gold machine are obtained, the first network service performance data and the second network service performance data are smoothed according to a preset processing rule, and the processed first network service performance data and the processed second network service performance data are used as the target network performance data;

[0012] If the performance evaluation type is a comprehensive network performance evaluation type, the first network performance data of the device under test and the second network performance data of the reference gold machine are obtained, the first time-space information corresponding to the first network performance data and the second time-space information corresponding to the second network performance data are determined, the first historical network performance data associated with the first time-space information and the second historical network performance data matching the second time-space information are obtained, and the first network performance data, the second network performance data, the first historical network performance data and the second historical network performance data are used as the target network performance data;

[0013] If the performance evaluation type is a batch network performance evaluation type, the basic network performance data of the devices under test included in the current batch is obtained, and the basic network performance data of the devices under test included in the current batch is used as the target network performance data.

[0014] Further, the acquiring of first historical network performance data associated with the first spatiotemporal information and second historical network performance data matching the second spatiotemporal information includes:

[0015] Acquire a network map data model, wherein the network map data model includes historical network performance data corresponding to the device under test at different historical locations and different historical times;

[0016] Extracting time information and spatial position information from the first spatiotemporal information;

[0017] Acquire a historical relative position matching the spatial position information from the network map data model, and acquire historical network performance data corresponding to different historical times associated with the historical relative position;

[0018] The first historical network performance data matching the time information is obtained from the historical network performance data corresponding to different historical times associated with the first historical relative position, and the second historical network performance data matching the second spatiotemporal information is obtained from the historical network performance data set corresponding to the reference gold machine.

[0019] Furthermore, the acquiring of the historical relative position matching the spatial position information from the network map data model includes:

[0020] Acquire potential historical location information adjacent to the spatial location information from the network map data model;

[0021] Calculating a position difference based on the spatial position information and the potential historical position information, and taking the potential historical position information whose position difference is less than a preset distance difference as candidate historical position information;

[0022] Calculate a first speed value according to a change in the distance between the spatial position information and the candidate historical position information and a first time difference, wherein the first time difference is calculated based on historical time information corresponding to the time information and the candidate historical position information;

[0023] The second speed is obtained by calculating according to the distance change between the candidate historical position information at different historical moments and the second time difference, wherein the second time difference is calculated by the candidate historical position information corresponding to different historical time information;

[0024] A speed difference between the first speed value and the second speed value is calculated, and the historical relative position is determined using candidate historical position information whose speed difference is less than a preset speed difference.

[0025] Furthermore, the method further comprises:

[0026] Performing correlation analysis on the first network performance data and the historical network performance data in the network map data model to obtain spatiotemporal characteristics and a plurality of correlation data between the first network performance data and the historical network performance data;

[0027] updating the associated data to a network map data model according to the spatiotemporal characteristics;

[0028] According to a preset algorithm in the network map data model, a plurality of associated data between the first network performance data and the historical network performance data are merged to obtain an optimized network map data model.

[0029] Further, the target network performance data is evaluated according to the target evaluation strategy to obtain a performance evaluation result corresponding to the device under test, including:

[0030] If the performance evaluation type is a real-time network basic coverage performance evaluation type, obtaining first indicator data corresponding to a preset indicator from the first network basic performance data, and obtaining second indicator data corresponding to the preset indicator from the second network basic data;

[0031] Comparing the first indicator data and the second indicator data associated with each of the preset indicators;

[0032] If there is at least one preset indicator associated with a first indicator data lower than the second indicator data, then the performance evaluation result is determined to be that the basic performance of the device under test is lower than the basic performance of the reference gold machine; or, if the first indicator data associated with the preset indicators are all higher than the second indicator data, then the performance evaluation result is determined to be that the basic performance of the device under test is higher than the basic performance of the reference gold machine.

[0033] Further, the target network performance data is evaluated according to the target evaluation strategy to obtain a performance evaluation result corresponding to the device under test, including:

[0034] If the performance evaluation type is a real-time network service performance evaluation type, obtaining third indicator data corresponding to a preset indicator from the first network service performance data, and obtaining fourth indicator data corresponding to the preset indicator from the second network service data;

[0035] Comparing the third indicator data and the fourth indicator data associated with each of the preset indicators;

[0036] If there is at least one preset indicator associated with a third indicator data lower than the fourth indicator data, then the performance evaluation result is determined to be that the business performance of the device under test is lower than the business performance of the reference gold machine; or, if the third indicator data associated with the preset indicators are all higher than the fourth indicator data, then the performance evaluation result is determined to be that the business performance of the device under test is higher than the business performance of the reference gold machine.

[0037] Further, the target network performance data is evaluated according to the target evaluation strategy to obtain a performance evaluation result corresponding to the device under test, including:

[0038] If the performance evaluation type is a comprehensive network performance evaluation type, obtaining the current performance of the device under test by comparing the first network performance data and the second network performance data;

[0039] Comparing the first historical network performance data with the second historical network performance data to obtain historical performance information of the device under test;

[0040] Comparing the first network performance data with the first historical network performance data to obtain a change trend of the network performance of the device under test over time;

[0041] Analyze the current performance and the historical performance to obtain performance differences of the device under test;

[0042] The performance evaluation result is generated based on the performance difference of the tested device and the change trend.

[0043] Further, the target network performance data is evaluated according to the target evaluation strategy to obtain a performance evaluation result corresponding to the device under test, including:

[0044] If the performance evaluation type is a batch network performance evaluation type, the network basic performance data of the devices under test in the current batch are processed according to the preset processing rules to obtain the processed network basic performance data of the devices under test in the current batch;

[0045] Obtain the time-space status information corresponding to the network basic performance data of the tested equipment in the current batch;

[0046] Using the spatiotemporal state information, obtain the corresponding associated network basic performance data of each device under test included in the current batch;

[0047] The network basic performance data of each device under test included in the current batch is compared with the associated network basic performance data to obtain a performance evaluation result corresponding to the device under test.

[0048] Furthermore, the obtaining of the associated network basic performance data corresponding to each device under test included in the current batch using the spatiotemporal state information includes:

[0049] Acquire a network map data model, wherein the network map data model includes historical network performance data corresponding to a plurality of devices under test at historical locations of different devices under test and at different historical times;

[0050] Obtaining a historical network performance data set corresponding to the tested devices included in the current batch from the network map data model;

[0051] The temporal and spatial state information is used to filter the corresponding associated network basic performance data of each tested device in the current batch from the historical network performance data set.

[0052] In a second aspect, an embodiment of the present invention provides a performance evaluation device for an intelligent connected vehicle, the device comprising:

[0053] An acquisition module, used to acquire a network performance evaluation instruction for an intelligent connected vehicle, wherein the network performance evaluation instruction includes a performance evaluation type and a target evaluation strategy;

[0054] A response module, used to respond to the network performance evaluation instruction, obtain target network performance data associated with the device under test in the intelligent connected vehicle using the performance evaluation type, and obtain a device operation log corresponding to the device under test;

[0055] The evaluation module is used to evaluate the target network performance data according to the target evaluation strategy, obtain the performance evaluation result corresponding to the device under test, and perform corresponding processing operations on the device operation log according to the performance evaluation result.

[0056] In a third aspect, an embodiment of the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0057] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method of the first aspect or any corresponding embodiment thereof.

[0058] The method provided in the embodiment of the present application clarifies the performance evaluation type and target evaluation strategy by obtaining network performance evaluation instructions for intelligent connected vehicles, which makes the test no longer limited to the previous single isolated road test data mode, and can obtain data from multiple aspects according to different evaluation types. Secondly, the response instruction uses the performance evaluation type to obtain the target network performance data and equipment operation log associated with the device under test, comprehensively covering all kinds of situation records during the operation of the equipment, thereby realizing a comprehensive quantitative record of network-related indicators, changing the previous situation of lack of comprehensive quantitative records. Furthermore, in the process of evaluating the target network performance data according to the target evaluation strategy to obtain the performance evaluation results, it will involve comparative analysis of data under different conditions, which makes up for the previous lack of horizontal and vertical data analysis.

[0059] By obtaining the corresponding first indicator data and second indicator data from the first network basic performance data and the second network basic data according to the preset indicators, it is possible to focus on the key evaluation factors, make the evaluation more targeted and accurate, and avoid the interference of irrelevant data. Secondly, by comparing the two sets of data associated with each preset indicator, the differences between the tested device and the reference gold machine in each key network basic performance indicator can be clearly shown. This one-by-one comparison method can deeply explore the differences in performance between the two. Finally, the high and low relationship between the basic performance of the tested device and the reference gold machine is determined based on the comparison results. Whether there is at least one indicator data that is low, indicating that the tested device has poor performance, or all indicator data are high, indicating that its performance is excellent, it can provide a clear conclusion for the network basic performance evaluation, which is helpful to quickly and accurately judge whether the tested device meets the standards or has advantages and disadvantages in terms of network basic performance, and then guide the subsequent optimization, improvement or selection and other related work.

[0060] When the performance evaluation type is a real-time network service performance evaluation type, the corresponding third indicator data and fourth indicator data are obtained from the first network service performance data and the second network service data according to the preset indicators, which can accurately locate the key service performance indicators, so that the evaluation focuses on the factors that have an important impact on the actual network service operation status, effectively eliminating the interference of irrelevant information, and making the evaluation more targeted. Second, the third indicator data and the fourth indicator data associated with each preset indicator are compared one by one, which can clearly and meticulously present the specific differences between the tested device and the reference gold machine in various key service performance indicators, so as to fully and deeply understand the different performances of the two in network service performance. Third, the high and low relationship between the service performance of the tested device and the reference gold machine is determined based on the comparison results. If there is at least one preset indicator data lower than the other party, it can be clearly known that the service performance of the tested device is poor. On the contrary, if all preset indicator data are higher than the other party, it means that its service performance is better. In this way, a definite conclusion can be given for the network service performance evaluation, which is helpful to quickly and accurately judge the advantages and disadvantages of the tested device in network service performance, and then provide a strong basis for subsequent optimization and adjustment of service performance, equipment selection and other related work.

[0061] When the performance evaluation type is a comprehensive network performance evaluation type, the current performance of the device under test is obtained by comparing the first network performance data and the second network performance data. This can intuitively present the performance of the device under test relative to the reference gold machine in various network performance indicators at the moment, and clearly grasp its operating status in the current network environment, including network speed, stability and other aspects of the real-time situation. Secondly, the first historical network performance data is compared with the second historical network performance data to obtain the historical performance of the device under test. This allows an in-depth understanding of the past network performance of the device under test from the time dimension, and insight into its performance change trend over time, such as whether there is a gradual decline or improvement in performance, etc., which provides an important historical reference for comprehensive evaluation. Finally, the current performance situation and the historical performance situation are comprehensively analyzed to obtain the corresponding performance evaluation results of the device under test. This comprehensive and comprehensive evaluation method can more accurately judge the overall network performance of the device under test. It not only takes into account the current performance, but also takes into account the historical development trend, so that potential performance problems can be discovered more accurately, and it provides a very reliable and comprehensive decision-making basis for subsequent targeted network optimization, equipment upgrades or adjustments, which helps to better ensure the efficient and stable operation of the network system. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0063] Figure 1 is a schematic flow chart of a method for evaluating the performance of an intelligent connected vehicle according to some embodiments of the present invention;

[0064] Figure 2 is a schematic diagram of a test system according to some embodiments of the present invention;

[0065] Figure 3 is a flow chart of another method for evaluating the performance of an intelligent connected vehicle according to some embodiments of the present invention;

[0066] Figure 4 is a flow chart of another method for evaluating the performance of an intelligent connected vehicle according to some embodiments of the present invention;

[0067] Figure 5 is a flow chart of another method for evaluating the performance of an intelligent connected vehicle according to some embodiments of the present invention;

[0068] Figure 6is a flow chart of another method for evaluating the performance of an intelligent connected vehicle according to some embodiments of the present invention;

[0069] Figure 7 is a structural block diagram of a performance evaluation device for an intelligent connected vehicle according to an embodiment of the present invention;

[0070] Figure 8 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0072] According to an embodiment of the present invention, a performance evaluation method, device, equipment and medium for an intelligent connected vehicle are provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0073] In this embodiment, a method for evaluating the performance of an intelligent connected vehicle is provided. Figure 1 is a flow chart of a performance evaluation method for an intelligent connected vehicle according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0074] Step S101, obtaining a network performance evaluation instruction for an intelligent connected vehicle, wherein the network performance evaluation instruction includes a performance evaluation type and a target evaluation strategy.

[0075] In the embodiments of the present application, Figure 2 As shown in the figure, both the device under test and the reference gold machine are connected to the control unit at the same time. This connection method has many possibilities. It can be through a physical connection method such as USB, and use tools such as adb to achieve data transmission and control functions; it can also be based on a virtual connection, such as a wireless personal area network (WPAN), local area network (LAN), Internet, etc., and can also use tools such as adb and apk to complete the corresponding operations. This provides a variety of options for testing in different environments and requirements, and the appropriate connection method can be flexibly selected according to the actual situation.

[0076] At the same time, the control unit will send instructions for the same action target to the reference golden machine and the device under test. This means that the task requirements given to the two are the same, so that their execution can be compared under the same conditions. Subsequently, the reference golden machine and the device under test each execute these instructions and return the results of the execution to the control unit, so that the control unit can obtain the completion data of the same task by both. When the adb tool based on USB physical connection is used, various control, query and other instructions will be accurately issued to the device under test and the reference golden machine according to the device ID number. The specific instruction format is such as "adb-s <device ID number of the device under test or reference golden machine> shell <encapsulated instruction>". In this way, the specific device can be accurately located and specific instruction content can be sent to it to ensure that the instruction can reach the target device accurately and be executed.

[0077] The control unit will also obtain the latitude and longitude information with a time stamp of the reference gold machine or the device under test. This latitude and longitude information is very critical, which is equivalent to marking the network performance data with time and space. By associating the network performance data with the acquired timestamp and the time and space represented by the longitude and latitude information, the results of the network test can be traceable. In other words, when you need to view or analyze the network performance at a specific time and place in the future, you can easily find the corresponding test results. At the same time, this also makes the test results easy to repeat (retest), because the test environment can be accurately restored according to the previously recorded time and space conditions; and it is easy to compare, such as comparing the network performance differences between different times, different places or different devices, so as to analyze the network performance status more comprehensively and in-depth.

[0078] In the embodiment of the present application, when the network performance of the intelligent connected vehicle is evaluated, the corresponding evaluation instruction will be obtained, which covers the performance evaluation type and the target evaluation strategy. The performance evaluation types include real-time network basic performance evaluation based on the reference gold machine, that is, comparing the network basic performance data such as RSSI, RSRP, SNR, RSRQ of the tested device and the reference gold machine at the same time point, combining data processing and the network, system, and application logs of the tested device to monitor; real-time network service performance evaluation based on the reference gold machine, by initiating service instructions to the tested device and the reference gold machine to obtain service performance data such as ping, upload, download, video playback, and time location information and post-processing comparison; the network performance (basic and business) history of the reference gold machine and the tested device - real-time comprehensive evaluation, which can compare the network basic and business performance data in real time, and can also call up the historical data with timestamps for comprehensive comparison on demand; the tested device's own network basic performance evaluation based on historical data, using the historical big database to compare the current network basic performance data of the tested device with its own historical records; large-scale network basic performance evaluation, for a large number of tested devices, using wireless network remote control to obtain network basic performance data and time location information and then compare with related data. The target evaluation strategy is based on different types. After data acquisition, processing and comparison, it is combined with log management, such as deciding log storage or deletion, triggering alarms, etc. based on the comparison results, to achieve effective evaluation of the network performance of intelligent connected vehicles.

[0079] Step S102, responding to the network performance evaluation instruction, using the performance evaluation type to obtain the target network performance data associated with the device under test in the intelligent connected vehicle, and obtaining the device operation log corresponding to the device under test.

[0080] In the embodiment of the present application, the target network performance data associated with the device under test in the intelligent connected vehicle is obtained by using the performance evaluation type, including the following situations:

[0081] ① If the performance evaluation type is a real-time network basic coverage performance evaluation type, the first network basic performance data of the device under test and the second network basic performance data of the reference gold machine are obtained, and the first network basic performance data and the second network basic performance data are smoothed according to the preset processing rules, and the processed first network basic performance data and the processed second network basic performance data are used as the target network performance data.

[0082] Specifically, a series of operation processes are taken when it is determined that the performance evaluation type is a real-time network basic coverage performance evaluation type. First, the relevant network basic performance data of the device under test should be obtained separately, which is referred to as the first network basic performance data here, and the network basic performance data of the reference gold machine as a reference standard should also be obtained, that is, the second network basic performance data. Then, according to the pre-set processing rules, the two sets of obtained network basic performance data, that is, the first network basic performance data and the second network basic performance data, are respectively smoothed. Finally, the two sets of processed data, that is, the processed first network basic performance data and the processed second network basic performance data, are determined as the target network performance data for subsequent further analysis, comparison and other operations, so as to carry out relevant evaluation work such as judging the difference between the device under test and the reference gold machine in network basic performance based on these target network performance data.

[0083] For example, data smoothing can be achieved by using a moving average method. Specifically, you can select a time period of 10 seconds and perform a moving average calculation on the relevant network basic performance data within the time period. By continuously sliding this time period to process the data of each time period in turn, the data fluctuation can be smoothed out to highlight the overall change trend. You can also perform a moving average based on a location distance difference of 100m, that is, perform a corresponding average calculation on the data within each location distance of 100m, and repeat this operation as the position changes in turn, so as to reduce the discreteness of the data and make the data smoother.

[0084] ② If the performance evaluation type is a real-time network service performance evaluation type, obtain the first network service performance data of the device under test and the second network service performance data of the reference gold machine, perform data smoothing on the first network service performance data and the second network service performance data according to the preset processing rules, and use the processed first network service performance data and the processed second network service performance data as the target network performance data.

[0085] Specifically, a series of operation processes when the performance evaluation type is determined to be a real-time network service performance evaluation type. Specifically, the first thing to do is to obtain the first network service performance data corresponding to the device under test, which involves indicators such as ping, upload, download, video playback, etc. that can reflect the actual operation status of the network service; at the same time, the second network service performance data of the reference gold machine must also be obtained. After that, according to the pre-set processing rules, the two sets of data obtained, namely the first network service performance data and the second network service performance data, are smoothed. Finally, the two sets of data obtained after processing, namely the processed first network service performance data and the processed second network service performance data, are determined as target network performance data, so that related work such as comparative analysis can be carried out based on these target network performance data in the future, so as to evaluate the performance of the device under test relative to the reference gold machine in terms of network service performance.

[0086] For example, data smoothing can be achieved by using a moving average method. Specifically, you can select a time period of 10 seconds and perform a moving average calculation on the relevant network basic performance data within the time period. By continuously sliding this time period to process the data of each time period in turn, the data fluctuation can be smoothed out to highlight the overall change trend. You can also perform a moving average based on a location distance difference of 100m, that is, perform a corresponding average calculation on the data within each location distance of 100m, and repeat this operation as the position changes in turn, so as to reduce the discreteness of the data and make the data smoother.

[0087] ③ If the performance evaluation type is a comprehensive network performance evaluation type, obtain the first network performance data of the device under test and the second network performance data of the reference gold machine, determine the first time-space information corresponding to the first network performance data and the second time-space information corresponding to the second network performance data, obtain the first historical network performance data associated with the first time-space information and the second historical network performance data matching the second time-space information, and use the first network performance data, the second network performance data, the first historical network performance data and the second historical network performance data as target network performance data.

[0088] Specifically, obtaining first historical network performance data associated with the first spatiotemporal information and second historical network performance data matching the second spatiotemporal information includes the following steps 1-7:

[0089] Step 1: Obtain a network map data model, wherein the network map data model includes historical network performance data corresponding to the device under test at different historical locations and different historical times.

[0090] Specifically, the existing network map data model is accessed through the corresponding data access interface. This model contains the historical network performance data corresponding to the device under test at different historical locations and different historical times. The collection of historical network performance data may be obtained by continuously monitoring the basic network performance data (RSSI / RSRP / SNR / RSRQ, etc.) and network service performance data (such as ping, upload, download, video playback, etc.) of the device under test at different times and locations in the past, and recording the corresponding spatiotemporal status information and then collating it. These data are organized according to geographic map information so that they can accurately reflect the network performance status at different locations and times.

[0091] Step 2: extracting time information and spatial position information from the first spatiotemporal information.

[0092] Specifically, after the first spatiotemporal information is obtained, the structure and content of the information are analyzed. A specific data extraction algorithm or tool is used to separate the time information part and the spatial location information part from the first spatiotemporal information. For the time information, specific information such as the date and time point can be extracted. For the spatial location information, it can be parsed according to different representation methods. For example, if it is a longitude and latitude coordinate, the corresponding longitude and latitude values ​​are extracted. Ensure that the extracted time information and spatial location information are accurate for use in subsequent steps.

[0093] Step 3: Obtain the historical relative position that matches the spatial position information from the network map data model, and obtain the historical network performance data corresponding to different historical times associated with the historical relative position.

[0094] Specifically, the historical relative position matching the spatial position information is obtained from the network map data model, including:

[0095] ① Obtain potential historical location information adjacent to the spatial location information from the network map data model.

[0096] Based on the positioning information and the derivative of the time difference of the adjacent position difference, select according to a certain error. Potential historical position information selection: According to the formula a1 = (N + H) cosb cosL, b1 = (N + H) cosB sinL, c1 = [N (1-e 2 )+H]sinB, where (L, B, H) are the latitude and longitude of the earth and the height of the earth respectively, (a1, b1, c1) is the position calculated to the spatial rectangular coordinate system, N is the radius of the ellipsoid circle, e is the first eccentricity of the ellipsoid, a and b are the major and minor axes of the Earth respectively.

[0097] ② Calculate the position difference based on the spatial position information and the potential historical position information, and use the potential historical position information whose position difference is less than the preset distance difference as the candidate historical position information.

[0098] In the spatial rectangular coordinate system (a1, b1, c1), (a2, b2, c2)....(a n ,b n ,c n ) is the current position center of the device under test according to time. n ,B n ,C n ) does not contain any historical location information, and the location difference is calculated by analogy, according to the distance formula between two points:

[0099] The relative position relationship between the current position and the historical position can be determined. When d is less than the preset distance difference, it is considered that the historical position can be preliminarily selected. The selected historical position is used as candidate historical position information.

[0100] ③ Calculate the first speed value based on the distance change between the spatial position information and the candidate historical position information and the first time difference, wherein the first time difference is calculated based on the historical time information corresponding to the time information and the candidate historical position information.

[0101] For the candidate historical position information, the speed formula of the first speed value is: t1 and t2 are the current time differences.

[0102] ④ Calculate the second speed based on the distance change between the candidate historical position information at different historical moments and the second time difference, wherein the second time difference is calculated based on the candidate historical position information corresponding to different historical time information;

[0103] Speed ​​formula for the second speed value: T1 and T2 are historical time differences.

[0104] ⑤ Calculate the speed difference between the first speed value and the second speed value, and determine the historical relative position using the candidate historical position information whose speed difference is less than the preset speed difference.

[0105] Specifically, ΔV=vV, when ΔV meets the set value, the network information of the historical position is considered valid. That is, the speed change corresponding to each historical relative position is analyzed, and the historical relative position whose speed difference meets the first speed difference setting requirement is selected.

[0106] In this way, by obtaining the potential historical position information adjacent to the spatial position information from the network map data model, the existing historical data can be fully utilized to assist in determining the relevant information of the current position. Next, the position difference is calculated and the potential historical position information whose position difference is less than the preset distance difference is selected as the candidate historical position information, which can narrow the analysis scope and improve the processing efficiency and accuracy. Then, the first speed value is calculated according to the distance change and time difference between the spatial position information and the candidate historical position information, and the second speed value is calculated according to the distance change and time difference between the candidate historical position information at different historical moments, so that the movement of the device under test in different time periods can be more comprehensively understood. Finally, by calculating the speed difference and determining the historical relative position using the candidate historical position information whose speed difference is less than the preset speed difference, the historical position related to the current position can be determined more accurately, providing a more reliable basis for the subsequent network performance analysis and the optimization of the network map data model, thereby better reflecting the network performance status at different times and places, and improving the accuracy and practicality of the network map data model.

[0107] Step 4, obtain the first historical network performance data matching the time information from the historical network performance data corresponding to different historical times associated with the first historical relative position, and obtain the second historical network performance data matching the second spatiotemporal information from the historical network performance data set corresponding to the reference gold machine.

[0108] Determine the storage location of the first historical relative position in the network map data model. Perform a matching search from the historical network performance data set corresponding to different historical times associated with the position based on the currently acquired time information. By comparing the proximity of each historical time to the current time information, select the historical network performance data closest to the current time as the first historical network performance data.

[0109] To obtain the second historical network performance data from the historical network performance data set corresponding to the reference gold machine, first parse the second spatiotemporal information and extract the time and space information therein. Then, in the historical network performance data set of the reference gold machine, match it according to the two dimensions of time and space. In the time dimension, find the historical record closest to the time in the second spatiotemporal information; in the space dimension, ensure that the position corresponding to the record has a certain correlation with the spatial position in the second spatiotemporal information. By combining the matching results of time and space, determine the historical network performance data that matches the second spatiotemporal information as the second historical network performance data.

[0110] ④ If the performance evaluation type is a batch network performance evaluation type, the basic network performance data of the devices under test included in the current batch is obtained, and the basic network performance data of the devices under test included in the current batch is used as the target network performance data.

[0111] Specifically, when the performance evaluation type is set to batch network performance evaluation type, the focus of the operation is to perform network performance evaluation on multiple devices under test in a batch. In this case, there is no need to compare with the reference gold machine or introduce complex operations such as historical data. Instead, the basic network performance data of all devices under test in the batch currently being processed are directly extracted. These data may include relevant indicators such as RSSI and RSRP that reflect the basic operation status of the network. Then, the basic network performance data of all devices under test in this batch are directly used as the target network performance data as a whole, so as to perform centralized network basic performance analysis and evaluation on the devices under test in the batch based on these data, so as to meet the needs of fast and overall grasp of network performance in large-scale testing or batch processing scenarios.

[0112] In an embodiment of the present application, the method also includes: performing correlation analysis on the first network performance data and the historical network performance data in the network map data model to obtain spatiotemporal characteristics and multiple correlation data between the first network performance data and the historical network performance data; updating the correlation data to the network map data model according to the spatiotemporal characteristics; and fusing the multiple correlation data between the first network performance data and the historical network performance data according to a preset algorithm in the network map data model to obtain an optimized network map data model.

[0113] First, the first network performance data corresponding to the device under test is correlated with the historical network performance data in the network map data model. In this process, by comparing information in dimensions such as time and space, spatiotemporal features are extracted, and multiple correlation data between the first network performance data and the historical network performance data are determined, such as the similarity and change trend of network performance parameters at a specific time and location.

[0114] Then, according to the extracted spatiotemporal features, these related data are updated to the network map data model. This means finding the corresponding time and location nodes in the network map data model, adding the new related data, and enriching the data content of the model.

[0115] Finally, according to the preset algorithm in the network map data model, multiple related data between the first network performance data and the historical network performance data are fused. It may be possible to use weighted average and other methods to determine the weight according to factors such as the importance and timeliness of the data, and to perform comprehensive calculations on different data to obtain an optimized network map data model, so that it can more accurately reflect the network performance status at different times and locations.

[0116] Step S103, evaluating the target network performance data according to the target evaluation strategy, obtaining the performance evaluation result corresponding to the device under test, and performing corresponding processing operations on the device operation log according to the performance evaluation result.

[0117] In the embodiment of the present application, the target network performance data is evaluated according to the target evaluation strategy to obtain the performance evaluation result corresponding to the device under test, such as Figure 3 As shown, the following steps A1-A3 are included:

[0118] Step A1: If the performance evaluation type is a real-time network basic coverage performance evaluation type, first indicator data corresponding to a preset indicator is obtained from the first network basic performance data, and second indicator data corresponding to the preset indicator is obtained from the second network basic data.

[0119] Specifically, when the performance evaluation type is determined to be a real-time network basic coverage performance evaluation type, the relevant data that has been obtained must be clarified first. In this scenario, there are first network basic performance data (these data should have been previously obtained from the device under test, including various indicators such as RSSI, RSRP, SNR, RSRQ, etc. that can reflect the basic performance of the network) and second network basic performance data (the same type of network basic performance indicator data obtained from the reference gold machine).

[0120] Then, for the pre-set key indicators for evaluation (i.e., preset indicators, such as RSSI and RSRP, which may be of particular concern as key evaluation basis), the specific data corresponding to these preset indicators are accurately extracted from the first network basic performance data, and defined as the first indicator data. Similarly, the corresponding second indicator data is accurately obtained from the second network basic performance data according to the requirements of the preset indicators. In this way, the operation of extracting the corresponding specific indicator data from the two sets of network basic performance data according to the preset indicators is completed, and preparations are made for subsequent comparative analysis.

[0121] Step A2: comparing the first indicator data and the second indicator data associated with each preset indicator.

[0122] Specifically, for each preset indicator, the first indicator data obtained from the device under test is compared with the second indicator data obtained from the reference gold machine. For example, if RSSI is included in the preset indicators, the first indicator data about RSSI extracted from the first network basic performance data is compared with the second indicator data about RSSI extracted from the second network basic performance data to check the relationship between the two values; the same is true for other preset indicators such as RSRP, SNR, RSRQ, etc., and the two sets of data corresponding to each preset indicator are carefully compared and analyzed in turn to clarify the performance differences between the device under test and the reference gold machine in each key indicator.

[0123] Step A3: If there is at least one preset indicator associated with a first indicator data lower than the second indicator data, the performance evaluation result is determined to be that the basic performance of the device under test is lower than the basic performance of the reference gold machine; or, if the first indicator data associated with the preset indicators are all higher than the second indicator data, the performance evaluation result is determined to be that the basic performance of the device under test is higher than the basic performance of the reference gold machine.

[0124] Specifically, if among all the preset indicators for comparison, it is found that the first indicator data associated with at least one preset indicator is lower than the corresponding second indicator data, this means that the performance of the device under test is not as good as the reference golden machine in these key indicators. At this time, it can be determined that the performance evaluation result is that the basic performance of the device under test is lower than the basic performance of the reference golden machine.

[0125] On the contrary, if after comparing one by one, it is found that the first indicator data associated with all preset indicators are higher than the corresponding second indicator data, this indicates that the performance of the tested device is better than that of the reference golden machine in each key indicator, and then it can be determined that the basic performance of the tested device is higher than that of the reference golden machine. In this way, the relationship between the tested device and the reference golden machine in terms of real-time network basic performance can be accurately determined based on the comparison of the preset indicator data.

[0126] In the embodiment of the present application, the target network performance data is evaluated according to the target evaluation strategy to obtain the performance evaluation result corresponding to the device under test, such as Figure 4 As shown, the following steps B1-B3 are included:

[0127] Step B1, if the performance evaluation type is a real-time network service performance evaluation type, third indicator data corresponding to the preset indicator is obtained from the first network service performance data, and fourth indicator data corresponding to the preset indicator is obtained from the second network service data.

[0128] Specifically, when it is determined that the performance evaluation type is a real-time network service performance evaluation type, the first thing to do is to clarify the relevant data that has been obtained. In this scenario, the first network service performance data and the second network service performance data have been obtained through corresponding operations. Among them, the first network service performance data is collected from the device under test, and contains a series of various indicator information such as ping, upload, download, video playback, etc. that can reflect the actual operating status of the device under test in terms of network services; the second network service performance data is obtained from the reference gold machine, and also covers similar indicators for measuring network service performance.

[0129] Next, for the pre-set key indicators for evaluation, that is, preset indicators (these preset indicators are usually determined based on specific evaluation needs and key business performance aspects, such as the round-trip time of ping, upload speed, download speed, smoothness of video playback, etc.), the specific data corresponding to these preset indicators are accurately extracted from the first network business performance data, and defined as the third indicator data. Similarly, the corresponding fourth indicator data is accurately obtained from the second network business performance data according to the requirements of the preset indicators. Through such operations, the corresponding specific indicator data is extracted from the two sets of network business performance data according to the preset indicators, which is ready for subsequent comparative analysis.

[0130] Step B2, comparing the third indicator data and the fourth indicator data associated with each preset indicator.

[0131] Specifically, after the third indicator data and the fourth indicator data corresponding to each preset indicator are obtained from the first network service performance data and the second network service performance data, the comparison operation is started.

[0132] For each preset indicator, the third indicator data obtained from the device under test is compared with the fourth indicator data obtained from the reference gold machine. For example, if the preset indicator is the round-trip time of ping, then the third indicator data about the round-trip time of ping extracted from the first network service performance data is compared with the fourth indicator data about the round-trip time of ping extracted from the second network service performance data to check the relationship between the two values; the same is true for other preset indicators such as upload speed, smoothness of video playback, etc., and the two sets of data corresponding to each preset indicator are carefully compared and analyzed in turn to clarify the performance differences between the device under test and the reference gold machine in each key indicator.

[0133] Step B3: If there is at least one third indicator data associated with a preset indicator that is lower than the fourth indicator data, then the performance evaluation result is determined to be that the business performance of the device under test is lower than the business performance of the reference gold machine; or, if the third indicator data associated with the preset indicators are all higher than the fourth indicator data, then the performance evaluation result is determined to be that the business performance of the device under test is higher than the business performance of the reference gold machine.

[0134] Specifically, if among all the preset indicators for comparison, it is found that the third indicator data associated with at least one preset indicator is lower than the corresponding fourth indicator data, this means that the performance of the device under test is not as good as the reference golden machine in these key indicators. At this time, it can be determined that the performance evaluation result is that the business performance of the device under test is lower than the business performance of the reference golden machine.

[0135] On the contrary, if after comparing one by one, it is found that the third indicator data associated with all preset indicators are higher than the corresponding fourth indicator data, this indicates that the performance of the tested device is better than that of the reference golden machine in each key indicator, and then it can be determined that the performance evaluation result is that the service performance of the tested device is higher than that of the reference golden machine. In this way, the relationship between the tested device and the reference golden machine in terms of real-time network service performance can be accurately determined based on the comparison of the preset indicator data.

[0136] In the embodiment of the present application, the target network performance data is evaluated according to the target evaluation strategy to obtain the performance evaluation result corresponding to the device under test, such as Figure 5 As shown, the following steps C1-C3 are included:

[0137] Step C1: If the performance evaluation type is a comprehensive network performance evaluation type, the current performance of the device under test is obtained by comparing the first network performance data and the second network performance data.

[0138] Specifically, when the performance evaluation type is a comprehensive network performance evaluation type, the control unit first obtains the first network performance data (including the current basic network performance data of the device under test such as RSSI, RSRP, SNR, RSRQ, etc. and network service performance data such as ping, upload, download, video playback, etc.) and the second network performance data (reference to the corresponding network performance data of the gold machine) based on a specific connection method. These data are all timestamped. Then, these data are processed according to the set rules, and the various indicators in the first network performance data and the second network performance data are directly compared, such as comparing the RSSI value and ping value of the device under test and the reference gold machine. Through comparative analysis of these indicators, the network performance status of the device under test relative to the reference gold machine at the current moment is determined, that is, the current performance status of the device under test is obtained.

[0139] Step C2, comparing the first historical network performance data and the second historical network performance data to obtain the historical performance of the device under test.

[0140] Specifically, when the performance evaluation type is a comprehensive network performance evaluation type, the control unit will, as needed, synchronously retrieve the first historical network performance data (historical network performance data of the device under test, covering data of network basic performance and network service performance-related indicators at different time points in the past) and the second historical network performance data (historical network performance data of the reference gold machine) of the associated time position from the designated location or the permanent area of ​​the storage unit. Afterwards, the corresponding indicators in the first historical network performance data and the second historical network performance data are compared and analyzed, such as checking the changes in the download speed of the device under test and the reference gold machine in a certain period of time in the past, so as to obtain the historical performance of the device under test.

[0141] Step C3, comparing the first network performance data with the first historical network performance data to obtain a time-dependent trend of the network performance of the device under test.

[0142] Specifically, a variety of methods can be used to analyze the changing trend. A common method is to draw a line graph with time as the horizontal axis and the selected network performance indicator value (such as bandwidth value) as the vertical axis. The data of the current and historical time points are marked on the graph, and the trend of the line can be visually judged by observing whether it is rising, falling, or maintaining a stable trend. The trend can also be quantitatively analyzed by calculating the rate of change of the performance indicator value between adjacent time points, such as calculating the growth rate or decrease rate of bandwidth in each time period. Such trend analysis operations are performed on various network performance indicators to fully understand the changing trend of the network performance of the tested device over time. For example, it is found that the network bandwidth has shown a gradual downward trend in recent months, while the packet loss rate has an upward trend, which indicates that the network performance is gradually deteriorating.

[0143] Step C4, analyzing the current performance and historical performance to obtain the performance difference of the device under test.

[0144] Specifically, for each network performance indicator, compare the corresponding data in the current performance with the historical performance. For example, check the difference between the current network bandwidth and the historical average bandwidth, or compare the fluctuation range of the current delay time compared with the same period in history. Not only pay attention to the difference in values, but also analyze the correlation between this difference at different times and different performance indicators. For example, it is found that not only the bandwidth has decreased, but also the delay has increased, and this situation is more serious than most of the time in history. This means that the device under test has a more significant difference in network performance, and there may be potential problems that affect network operation. Through a comprehensive comparative analysis of various indicators, the complete performance differences of the equipment under test are sorted out, and it is clear which performance indicators have changed significantly and the degree of change.

[0145] Step C5: generating a performance evaluation result based on the performance differences and change trends of the devices under test.

[0146] Specifically, the specific advantages and disadvantages of the current network performance compared with the historical performance in various indicators are clarified based on the performance differences of the tested equipment. At the same time, based on the trend of network performance changes over time, it is determined whether this change is a short-term fluctuation or a long-term stable trend, as well as the synergy of changes between different performance indicators. For example, if the performance difference shows that the current bandwidth has dropped significantly and the trend of change shows that this drop has lasted for a long time, then it is necessary to focus on bandwidth-related influencing factors.

[0147] Based on the above comprehensive analysis, the performance evaluation results are generated. The evaluation results include an overall evaluation of the network performance of the tested device (such as good, average, poor, etc.), specific network performance indicators with problems and abnormal conditions (such as insufficient bandwidth, high latency, severe packet loss, etc.), and some preliminary suggestions based on the analysis, such as whether the equipment needs to be inspected and repaired, and whether the network configuration needs to be optimized and adjusted.

[0148] In the embodiment of the present application, the target network performance data is evaluated according to the target evaluation strategy to obtain the performance evaluation result corresponding to the device under test, such as Figure 6 As shown, the following steps D1-D4 are included:

[0149] Step D1, if the performance evaluation type is a batch network performance evaluation type, the basic network performance data of the devices under test in the current batch are processed according to preset processing rules to obtain the processed basic network performance data of the devices under test in the current batch.

[0150] Specifically, when the performance evaluation type is a batch network performance evaluation type, the control unit first obtains the network basic performance data (including indicators such as RSSI, RSRP, SNR, RSRQ, etc.) of all the devices under test in the current batch based on the connection mode of remote control with a large number of devices under test through a wireless network. Then, the obtained network basic performance data are processed according to the preset processing rules. For example, according to the set rules of moving average with a time period of 10 seconds, or moving average with a position distance difference of 100m, the RSSI, RSRP and other network basic performance data of each device under test are processed accordingly. Through such a processing process, the processed network basic performance data of the devices under test in the current batch are finally obtained, so as to provide a more accurate and targeted data basis for subsequent evaluation and analysis.

[0151] Step D2, obtaining the spatiotemporal state information corresponding to the network basic performance data of the tested equipment in the current batch.

[0152] Specifically, the time-space state information corresponding to the basic network performance data of the tested devices in the current batch is obtained. Based on the same wireless network remote control connection method, the control unit obtains the corresponding time and location information data while obtaining the basic network performance data. These time and location information data constitute the time-space state information corresponding to the basic network performance data. It clarifies the basic network performance data of each tested device at a specific spatial location at a specific time, providing key time-space dimension information for further correlation analysis.

[0153] Step D3, using the spatiotemporal state information to obtain the corresponding associated network basic performance data of each device under test included in the current batch.

[0154] Specifically, the spatial-temporal status information is used to obtain the corresponding associated network basic performance data of each device under test included in the current batch, including: obtaining a network map data model, wherein the network map data model includes historical network performance data corresponding to multiple devices under test at historical positions of different devices under test and at different historical times; obtaining a historical network performance data set corresponding to the devices under test included in the current batch from the network map data model; and using the spatial-temporal status information to filter the corresponding associated network basic performance data of each device under test in the current batch from the historical network performance data set.

[0155] First, obtain the network map data model. This model stores the historical network performance data corresponding to multiple devices under test at different historical locations and different historical times. It may be obtained by continuously collecting and sorting the basic network performance data (such as RSSI / RSRP / SNR / RSRQ, etc.) and network service performance data (such as ping, upload, download, video playback, etc.) of each device under test in different time and space states, and organizing them according to geographic map information and time.

[0156] Next, from this network map data model, according to the needs of the current batch, the historical network performance data set corresponding to the devices under test included in the current batch is obtained. This step can be done by querying and filtering, according to the identification of the device under test or other specific conditions, to extract the historical network performance data set related to the current batch from the model.

[0157] Finally, the spatiotemporal state information of each device under test in the current batch is used to filter from the historical network performance data set. Specifically, the time information and spatial location information are matched with the data in the historical network performance data set to find the basic network performance data associated with each device under test at a specific time and location, so that these data can be analyzed and processed later to provide a basis for network performance evaluation and optimization of the current batch.

[0158] Step D4, comparing the network basic performance data of each device under test contained in the current batch with the associated network basic performance data to obtain a performance evaluation result corresponding to the device under test.

[0159] Specifically, after obtaining the processed network basic performance data of the tested device in the current batch and the corresponding associated network basic performance data, the comparison operation begins. The processed network basic performance data is compared with the associated network basic performance data one by one according to various indicators, such as the values ​​of indicators such as RSSI and RSRP. If it is found that the processed network basic performance data is lower than the associated network basic performance data in some indicators, it means that under the corresponding time and space conditions, the network basic performance of the tested device is relatively poor; conversely, if the processed network basic performance data is higher than the associated network basic performance data in all indicators, it means that the network basic performance of the tested device under the time and space conditions is good. Through such a comprehensive comparative analysis, the corresponding performance evaluation results of the tested device are finally obtained, so that a clear judgment can be made on the network basic performance status of the tested device in the current batch, so as to take subsequent corresponding measures, such as optimizing and adjusting the poorly performing devices.

[0160] In the embodiment of the present application, the corresponding processing operations performed on the device operation log according to the performance evaluation results include the following situations:

[0161] ① After completing the comparative evaluation of the network basic performance data of the tested device and the reference golden machine, if the tested device is judged to be inferior to the reference golden machine, the corresponding action will be triggered. The network, system, application and other logs of the tested device in this period are simultaneously saved in the permanent area of ​​the storage unit or uploaded to the specified location, so as to further analyze the specific operating conditions and possible problems of the device during this poor performance. On the contrary, if it is judged that the tested device is not inferior to the reference golden machine, the network, system and application logs in the temporary area of ​​the storage unit are deleted cyclically according to the set deletion rules, so as to save the storage unit space and avoid unnecessary log accumulation.

[0162] ② After initiating business instructions to the device under test and the reference gold machine to obtain relevant data, and after data processing and comparing the network business performance data of the two, if it is determined that the network business performance of the device under test is worse than that of the reference gold machine, then an alarm action will be triggered, and the network, system, application and other logs of the device under test in this period will be saved in the permanent area of ​​the storage unit or uploaded to a designated location, so as to facilitate subsequent in-depth investigation of the reasons that may lead to poor business performance. If the network business performance of the device under test is not worse than that of the reference gold machine, the network, system, and application logs in the temporary area of ​​the storage unit will be deleted cyclically according to the set deletion rules to save storage unit space and ensure the effective use of storage resources.

[0163] ③After completing the comprehensive comparison of real-time and historical network performance data, process the equipment operation log based on the evaluation results. If it is found that there are obvious problems with the network performance of the tested device at the current or historical specific time location, such as poor performance compared to the reference gold machine or a significant downward trend in its own historical data, the network, system, application and other logs of the tested device for the corresponding period will be saved in the permanent area of ​​the storage unit or uploaded to a designated location for subsequent detailed analysis of these problem periods. On the contrary, if the evaluation results show that the overall network performance of the device is relatively stable and there are no obvious abnormalities, these logs in the temporary area of ​​the storage unit will be deleted cyclically according to the set deletion rules to save storage unit space.

[0164] ④ After comparing and evaluating the current network basic performance data of the tested device with its own historical record information, if it is determined that the device performance has changed, such as performance degradation, the network, system, application and other logs of the tested device during this period will be saved in the permanent area of ​​the storage unit or uploaded to a designated location for in-depth analysis of the reasons for the performance change. If the comparison results show that the device performance is relatively stable and there are no obvious abnormalities, then according to the set deletion rules, the network, system, and application logs in the temporary area of ​​the storage unit will be deleted cyclically to save storage unit space and maintain the efficiency of log management.

[0165] ⑤ After evaluating the basic network performance of a large number of tested devices, after data processing and comparison with the basic network performance data at the relevant time position, an alarm action will be triggered for those devices that are judged to have poor performance, and the network, system, application and other logs of the device in this period will be saved in a specific area of ​​its internal storage space, so as to facilitate subsequent analysis and troubleshooting of these poor performance devices. For devices with normal performance, according to the set deletion rules, the network, system, and application logs in the temporary area of ​​the internal storage space of these devices will be deleted cyclically to save the internal storage space of the device and ensure that the storage resources can meet the storage needs during large-scale testing.

[0166] The method provided in the embodiment of the present application clarifies the performance evaluation type and target evaluation strategy by obtaining network performance evaluation instructions for intelligent connected vehicles, which makes the test no longer limited to the previous single isolated road test data mode, and can obtain data from multiple aspects according to different evaluation types. Secondly, the response instruction uses the performance evaluation type to obtain the target network performance data and equipment operation log associated with the device under test, comprehensively covering all kinds of situation records during the operation of the equipment, thereby realizing a comprehensive quantitative record of network-related indicators, changing the previous situation of lack of comprehensive quantitative records. Furthermore, in the process of evaluating the target network performance data according to the target evaluation strategy to obtain the performance evaluation results, it will involve comparative analysis of data under different conditions, which makes up for the previous lack of horizontal and vertical data analysis.

[0167] In this embodiment, a performance evaluation device for an intelligent networked vehicle is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. As used below, the term "module" may be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0168] This embodiment provides a performance evaluation device for an intelligent connected vehicle, such as Figure 7 As shown, including:

[0169] An acquisition module 801 is used to acquire a network performance evaluation instruction for an intelligent connected vehicle, wherein the network performance evaluation instruction includes a performance evaluation type and a target evaluation strategy;

[0170] A response module 802 is used to respond to the network performance evaluation instruction, obtain the target network performance data associated with the device under test in the intelligent connected vehicle by using the performance evaluation type, and obtain the device operation log corresponding to the device under test;

[0171] The evaluation module 803 is used to evaluate the target network performance data according to the target evaluation strategy, obtain the performance evaluation result corresponding to the device under test, and perform corresponding processing operations on the device operation log according to the performance evaluation result.

[0172] In the embodiment of the present application, the response module 802 is used to obtain the first network basic performance data of the device under test and the second network basic performance data of the reference gold machine if the performance evaluation type is a real-time network basic coverage performance evaluation type, perform data smoothing on the first network basic performance data and the second network basic performance data according to a preset processing rule, and use the processed first network basic performance data and the processed second network basic performance data as the target network performance data; if the performance evaluation type is a real-time network service performance evaluation type, obtain the first network service performance data of the device under test and the second network service performance data of the reference gold machine, perform data smoothing on the first network service performance data and the second network service performance data according to a preset processing rule, and use the processed first network service performance data and the processed second network service performance data as the target network performance data. The business performance data is used as the target network performance data; if the performance evaluation type is a comprehensive network performance evaluation type, the first network performance data of the device under test and the second network performance data of the reference gold machine are obtained, the first time-space information corresponding to the first network performance data and the second time-space information corresponding to the second network performance data are determined, the first historical network performance data associated with the first time-space information and the second historical network performance data matching the second time-space information are obtained, and the first network performance data, the second network performance data, the first historical network performance data and the second historical network performance data are used as the target network performance data; if the performance evaluation type is a batch network performance evaluation type, the network basic performance data of the device under test contained in the current batch is obtained, and the network basic performance data of the device under test contained in the current batch is used as the target network performance data.

[0173] In an embodiment of the present application, response module 802 is used to obtain a network map data model, wherein the network map data model includes historical network performance data corresponding to the tested device at different historical positions and different historical times; extract time information and spatial position information from the first spatiotemporal information; obtain a historical relative position matching the spatial position information from the network map data model, and obtain historical network performance data corresponding to different historical times associated with the historical relative position; obtain first historical network performance data matching the time information from the historical network performance data corresponding to different historical times associated with the first historical relative position, and obtain second historical network performance data matching the second spatiotemporal information from the historical network performance data set corresponding to the reference gold machine.

[0174] In an embodiment of the present application, the response module 802 is used to obtain potential historical location information adjacent to the spatial location information from the network map data model; calculate the position difference based on the spatial location information and the potential historical location information, and use the potential historical location information whose position difference is less than a preset distance difference as the candidate historical location information; calculate according to the distance change between the spatial location information and the candidate historical location information and the first time difference to obtain a first speed value, wherein the first time difference is calculated between the time information and the historical time information corresponding to the candidate historical location information; calculate according to the distance change between the candidate historical location information at different historical moments and the second time difference to obtain a second speed, wherein the second time difference is calculated for different historical time information corresponding to the candidate historical location information; calculate the speed difference between the first speed value and the second speed value, and determine the historical relative position using the candidate historical location information whose speed difference is less than the preset speed difference.

[0175] In an embodiment of the present application, the device also includes: an updating module, which is used to perform correlation analysis on the first network performance data and the historical network performance data in the network map data model to obtain spatiotemporal characteristics and multiple correlation data between the first network performance data and the historical network performance data; update the correlation data to the network map data model according to the spatiotemporal characteristics; and fuse the multiple correlation data between the first network performance data and the historical network performance data according to a preset algorithm in the network map data model to obtain an optimized network map data model.

[0176] In an embodiment of the present application, the response module 802 is used to obtain a first historical network data set and a second historical network data set, wherein the first historical network data set includes historical network performance data corresponding to different historical times, and the second historical network data set includes historical network performance data corresponding to different positions; extract the first time and the first spatial position from the first spatiotemporal information, and extract the second time and the second spatial position from the second spatiotemporal information; obtain the first historical time that matches the time characteristics of the first time, and the second historical time that matches the time characteristics of the second time, and obtain the first candidate network performance data corresponding to the first historical time and the second candidate network performance data corresponding to the second historical time from the first original data set; obtain multiple historical relative positions associated with the first spatial position and multiple historical relative positions associated with the second spatial position, wherein the first spatial position The multiple historical relative positions associated with the first spatial position are screened based on the position difference between the first spatial position and the multiple historical relative positions associated with the second spatial position are screened based on the position difference between the second spatial position and the second spatial position; the first historical relative position is obtained from the multiple historical relative positions associated with the first spatial position using the first speed difference, and the second historical relative position is obtained from the multiple historical relative positions associated with the second spatial position using the second speed difference; the third candidate network performance data corresponding to the first historical relative position and the fourth candidate network performance data corresponding to the second historical relative position are obtained from the second original data set; the first historical network performance data associated with the first spatiotemporal information are constructed based on the first candidate network performance data and the third candidate network performance data, and the second historical network performance data associated with the second spatiotemporal information are constructed based on the second candidate network performance data and the fourth candidate network performance data.

[0177] In an embodiment of the present application, the evaluation module 803 is used to obtain the first indicator data corresponding to the preset indicator from the first network basic performance data, and obtain the second indicator data corresponding to the preset indicator from the second network basic data if the performance evaluation type is a real-time network basic coverage performance evaluation type; compare the first indicator data and the second indicator data associated with each preset indicator; if there is at least one first indicator data associated with the preset indicator that is lower than the second indicator data, determine that the performance evaluation result is that the basic performance of the device under test is lower than the basic performance of the reference gold machine; or, if the first indicator data associated with the preset indicators are all higher than the second indicator data, determine that the performance evaluation result is that the basic performance of the device under test is higher than the basic performance of the reference gold machine.

[0178] In an embodiment of the present application, the evaluation module 803 is used to obtain the third indicator data corresponding to the preset indicator from the first network service performance data, and obtain the fourth indicator data corresponding to the preset indicator from the second network service data if the performance evaluation type is a real-time network service performance evaluation type; compare the third indicator data and the fourth indicator data associated with each preset indicator; if there is at least one third indicator data associated with the preset indicator that is lower than the fourth indicator data, determine that the performance evaluation result is that the service performance of the device under test is lower than the service performance of the reference gold machine; or, if the third indicator data associated with the preset indicators are all higher than the fourth indicator data, determine that the performance evaluation result is that the service performance of the device under test is higher than the service performance of the reference gold machine.

[0179] In an embodiment of the present application, the evaluation module 803 is used to, if the performance evaluation type is a comprehensive network performance evaluation type, obtain the current performance of the device under test by comparing the first network performance data and the second network performance data; obtain the historical performance of the device under test by comparing the first historical network performance data and the second historical network performance data; obtain the changing trend of the network performance of the device under test over time by comparing the first network performance data and the first historical network performance data; analyze the current performance and the historical performance to obtain the performance difference of the device under test; and generate a performance evaluation result based on the performance difference and the changing trend of the device under test.

[0180] In the embodiment of the present application, the evaluation module 803 is used to process the network basic performance data of the devices under test in the current batch according to the preset processing rules if the performance evaluation type is a batch network performance evaluation type, and obtain the processed network basic performance data of the devices under test in the current batch; obtain the spatiotemporal state information corresponding to the network basic performance data of the devices under test in the current batch;

[0181] The time-space state information is used to obtain the corresponding associated network basic performance data of each device under test included in the current batch; the network basic performance data of each device under test included in the current batch is compared with the associated network basic performance data to obtain the corresponding performance evaluation result of the device under test.

[0182] In an embodiment of the present application, the evaluation module 803 is used to obtain a network map data model, wherein the network map data model includes historical network performance data corresponding to multiple devices under test at historical positions of different devices under test and at different historical times; obtain a historical network performance data set corresponding to the devices under test included in the current batch from the network map data model; and use the spatiotemporal state information to filter the corresponding associated network basic performance data of each device under test in the current batch from the historical network performance data set.

[0183] See also Figure 8 , Figure 8is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 8 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8 A processor 10 is taken as an example.

[0184] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0185] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0186] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the use of a computer device based on the presentation of a small program landing page, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0187] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0188] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0189] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0190] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A performance evaluation method for an intelligent connected vehicle, characterized in that: The method comprises: Obtaining a network performance evaluation instruction for an intelligent connected vehicle, wherein the network performance evaluation instruction includes a performance evaluation type and a target evaluation strategy; In response to the network performance evaluation instruction, using the performance evaluation type to obtain target network performance data associated with the device under test in the intelligent connected vehicle, and obtaining a device operation log corresponding to the device under test; The target network performance data is evaluated according to the target evaluation strategy to obtain a performance evaluation result corresponding to the device under test, and corresponding processing operations are performed on the device operation log according to the performance evaluation result.

2. The method according to claim 1, characterized in that The obtaining target network performance data associated with the device under test in the intelligent connected vehicle by using the performance evaluation type includes: If the performance evaluation type is a real-time network basic coverage performance evaluation type, the first network basic performance data of the device under test and the second network basic performance data of the reference gold machine are obtained, the first network basic performance data and the second network basic performance data are smoothed according to a preset processing rule, and the processed first network basic performance data and the processed second network basic performance data are used as the target network performance data; If the performance evaluation type is a real-time network service performance evaluation type, the first network service performance data of the device under test and the second network service performance data of the reference gold machine are obtained, the first network service performance data and the second network service performance data are smoothed according to a preset processing rule, and the processed first network service performance data and the processed second network service performance data are used as the target network performance data; If the performance evaluation type is a comprehensive network performance evaluation type, the first network performance data of the device under test and the second network performance data of the reference gold machine are obtained, the first time-space information corresponding to the first network performance data and the second time-space information corresponding to the second network performance data are determined, the first historical network performance data associated with the first time-space information and the second historical network performance data matching the second time-space information are obtained, and the first network performance data, the second network performance data, the first historical network performance data and the second historical network performance data are used as the target network performance data; If the performance evaluation type is a batch network performance evaluation type, the basic network performance data of the devices under test included in the current batch is obtained, and the basic network performance data of the devices under test included in the current batch is used as the target network performance data.

3. The method according to claim 2, characterized in that The acquiring first historical network performance data associated with the first spatiotemporal information and second historical network performance data matching the second spatiotemporal information includes: Acquire a network map data model, wherein the network map data model includes historical network performance data corresponding to the device under test at different historical locations and different historical times; Extracting time information and spatial position information from the first spatiotemporal information; Acquire a historical relative position matching the spatial position information from the network map data model, and acquire historical network performance data corresponding to different historical times associated with the historical relative position; The first historical network performance data matching the time information is obtained from the historical network performance data corresponding to different historical times associated with the first historical relative position, and the second historical network performance data matching the second spatiotemporal information is obtained from the historical network performance data set corresponding to the reference gold machine.

4. The method according to claim 3, characterized in that The acquiring, from the network map data model, a historical relative position matching the spatial position information comprises: Acquire potential historical location information adjacent to the spatial location information from the network map data model; Calculating a position difference based on the spatial position information and the potential historical position information, and taking the potential historical position information whose position difference is less than a preset distance difference as candidate historical position information; Calculate a first speed value according to a change in the distance between the spatial position information and the candidate historical position information and a first time difference, wherein the first time difference is calculated based on historical time information corresponding to the time information and the candidate historical position information; A second speed value is obtained by calculating according to a distance change between different historical moments of the candidate historical position information and a second time difference, wherein the second time difference is calculated when the candidate historical position information corresponds to different historical time information; A speed difference between the first speed value and the second speed value is calculated, and the historical relative position is determined using candidate historical position information whose speed difference is less than a preset speed difference.

5. The method according to claim 3, characterized in that: The method further comprises: Performing correlation analysis on the first network performance data and the historical network performance data in the network map data model to obtain spatiotemporal characteristics and a plurality of correlation data between the first network performance data and the historical network performance data; updating the associated data to a network map data model according to the spatiotemporal characteristics; According to a preset algorithm in the network map data model, a plurality of associated data between the first network performance data and the historical network performance data are merged to obtain an optimized network map data model.

6. The method according to claim 2, characterized in that The step of evaluating the target network performance data according to the target evaluation strategy to obtain a performance evaluation result corresponding to the device under test includes: If the performance evaluation type is a real-time network basic coverage performance evaluation type, obtaining first indicator data corresponding to a preset indicator from the first network basic performance data, and obtaining second indicator data corresponding to the preset indicator from the second network basic data; Comparing the first indicator data and the second indicator data associated with each of the preset indicators; If there is at least one preset indicator associated with a first indicator data lower than the second indicator data, then the performance evaluation result is determined to be that the basic performance of the device under test is lower than the basic performance of the reference gold machine; or, if the first indicator data associated with the preset indicators are all higher than the second indicator data, then the performance evaluation result is determined to be that the basic performance of the device under test is higher than the basic performance of the reference gold machine.

7. The method according to claim 2, characterized in that The step of evaluating the target network performance data according to the target evaluation strategy to obtain a performance evaluation result corresponding to the device under test includes: If the performance evaluation type is a real-time network service performance evaluation type, obtaining third indicator data corresponding to a preset indicator from the first network service performance data, and obtaining fourth indicator data corresponding to the preset indicator from the second network service data; Comparing the third indicator data and the fourth indicator data associated with each of the preset indicators; If there is at least one preset indicator associated with a third indicator data lower than the fourth indicator data, then the performance evaluation result is determined to be that the business performance of the device under test is lower than the business performance of the reference gold machine; or, if the third indicator data associated with the preset indicators are all higher than the fourth indicator data, then the performance evaluation result is determined to be that the business performance of the device under test is higher than the business performance of the reference gold machine.

8. The method according to claim 3, characterized in that The target network performance data is evaluated according to the target evaluation strategy to obtain a performance evaluation result corresponding to the device under test, including If the performance evaluation type is a comprehensive network performance evaluation type, obtaining the current performance of the device under test by comparing the first network performance data and the second network performance data; Comparing the first historical network performance data with the second historical network performance data to obtain historical performance information of the device under test; Comparing the first network performance data with the first historical network performance data to obtain a change trend of the network performance of the device under test over time; Analyze the current performance and the historical performance to obtain performance differences of the device under test; The performance evaluation result is generated based on the performance difference of the tested device and the change trend.

9. The method according to claim 2, characterized in that: The step of evaluating the target network performance data according to the target evaluation strategy to obtain a performance evaluation result corresponding to the device under test includes: If the performance evaluation type is a batch network performance evaluation type, the network basic performance data of the devices under test in the current batch are processed according to the preset processing rules to obtain the processed network basic performance data of the devices under test in the current batch; Obtain the time-space status information corresponding to the network basic performance data of the tested equipment in the current batch; Using the spatiotemporal state information, obtain the corresponding associated network basic performance data of each device under test included in the current batch; The network basic performance data of each device under test included in the current batch is compared with the associated network basic performance data to obtain a performance evaluation result corresponding to the device under test.

10. The method according to claim 9, characterized in that The obtaining of the associated network basic performance data corresponding to each device under test included in the current batch by using the spatiotemporal state information includes: Acquire a network map data model, wherein the network map data model includes historical network performance data corresponding to a plurality of tested devices at historical locations of different tested devices and at different historical times; Obtaining a historical network performance data set corresponding to the tested devices included in the current batch from the network map data model; The temporal and spatial state information is used to filter the corresponding associated network basic performance data of each tested device in the current batch from the historical network performance data set.

11. A performance evaluation device for an intelligent connected vehicle, characterized in that: The device comprises: An acquisition module, used to acquire a network performance evaluation instruction for an intelligent connected vehicle, wherein the network performance evaluation instruction includes a performance evaluation type and a target evaluation strategy; A response module, used to respond to the network performance evaluation instruction, obtain target network performance data associated with the device under test in the intelligent connected vehicle using the performance evaluation type, and obtain a device operation log corresponding to the device under test; The evaluation module is used to evaluate the target network performance data according to the target evaluation strategy, obtain the performance evaluation result corresponding to the device under test, and perform corresponding processing operations on the device operation log according to the performance evaluation result.

12. An electronic device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method according to any one of claims 1 to 10 by executing the computer instructions.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 10.

Images