Intelligent equipment environment adaptability assessment method based on simulation environment

Abstract

The invention provides an intelligent equipment environment adaptability assessment method based on a simulation environment, which adopts a means of constructing a virtual simulation environment, generates different scene data from three dimensions of weather configuration, working condition configuration and interference configuration, then calculates scores of the intelligent equipment for completing tasks in different scenes according to an index system, and finally determines the environment adaptability of the intelligent equipment according to the scores. Therefore, the environment adaptability of the intelligent equipment is evaluated; the weather configuration is used for providing a virtual environment under various weather conditions, the working condition configuration is used for providing a simulated road condition in a verification environment, a scene data stream formed by the virtual environment under various weather conditions and the simulated road condition in the verification environment is in a video format, each frame of a video is a picture, and each frame of the video is a picture. And the interference configuration is used for simulating disturbance of electromagnetic interference on picture formation.

Description

Technical Field

[0001] The present invention relates to the field of testing, and in particular to an evaluation method for environmental adaptability of smart devices. Background Art

[0002] Environmental adaptability is an important quality characteristic of equipment, but the description of environmental adaptability at home and abroad is currently limited to the conceptual level. The usual practice is to qualitatively express whether it can be done through certain environmental tests, which cannot describe the adaptability of the equipment to its environment as a whole. Therefore, the environmental adaptability requirements cannot be truly implemented in the demonstration, design and production of the equipment, nor can they be accurately predicted, verified and evaluated. During the life of the equipment, the equipment will be exposed to various natural environments and induced environments during storage, transportation and use, which involve various environmental factors. Due to the different ways of characterizing the intensity of different environmental factors and their different mechanisms and speeds of action on the equipment, it becomes very complicated to characterize the environmental adaptability of the equipment. At present, it cannot be expressed by some simple parameters like reliability, such as mean time between failures and reliability. This is also an important reason affecting the development of the equipment environment engineering profession.

[0003] With the rapid development of artificial intelligence technology, various application elements are further digitized, networked, and intelligentized. The application scenarios of smart devices are rapidly expanding, and intelligent confrontation based on artificial intelligence has entered a new stage of development. Whether smart devices are safe and reliable in different application scenarios and whether they can maintain a high level of performance are issues that need to be confirmed in the actual application of smart devices. If smart devices that have not undergone environmental adaptability assessment are put into actual applications, their application capabilities will not be guaranteed, and it will be impossible to achieve reasonable personnel and equipment configuration, and it will be impossible to control potential risks. It may even be impossible to complete the designated application tasks, resulting in equipment damage and casualties.

[0004] Smart devices play an increasingly important role in today's competition, but due to the particularity of real-world scenarios, uncontrollable security issues are very likely to occur during the competition process. At present, the environmental adaptability test of smart devices is insufficient and cannot guarantee the smooth completion of tasks. Therefore, it is necessary to study the evaluation method of the environmental adaptability of smart devices to provide a strong guarantee for the research and development of smart devices, which is of great significance to seize the initiative in intelligent confrontation. Summary of the invention

[0005] In view of this, the present invention proposes a method for evaluating the environmental adaptability of intelligent devices based on a simulation environment. The method uses the means of constructing a virtual simulation environment to simulate four elements: basic natural environment (such as terrain, landform, water body, vegetation, etc.), different weather (such as sunny, cloudy, rainy, snowy, foggy, etc.), artificial scenes (such as man-made objects according to the perspective, subject movement mode, scene specificity, etc.), and anti-interference (such as simulating the imaging results of the intelligent device sensors being interfered with in the real scene), thereby forming an intelligent device environmental adaptability test environment, and then combining with the indicator calculation module to realize the environmental adaptability evaluation of the intelligent device. The present invention solves the problem of insufficient environmental adaptability testing of intelligent devices for target tracking and identification in different scenarios.

[0006] The specific technical scheme is as follows: a method for evaluating the environmental adaptability of intelligent devices based on a simulation environment, which adopts the means of constructing a virtual simulation environment to generate different scenario data from three dimensions: weather configuration, working condition configuration, and interference configuration, and then calculates the scores of the intelligent devices in completing tasks in different scenarios based on the indicator system, thereby completing the evaluation of the environmental adaptability of the intelligent devices; the weather configuration is used to provide a virtual environment under a variety of weather conditions, and the working condition configuration is used to provide simulated road conditions in a verification environment. The scene data stream composed of the virtual environment under a variety of weather conditions and the simulated road conditions in the verification environment is in video format, and each frame of the video is a picture. The interference configuration is used to simulate the disturbance caused by electromagnetic interference to the picture.

[0007] Furthermore, the weather configuration is realized by six aspects: meteorological data, sensor simulation, visual effect simulation, weather dynamics, and mission scenario integration. The working condition configuration is realized by random combination of different road sections to achieve different road conditions. The interference configuration is realized by six aspects: noise design, color distortion, image jitter, image blur, brightness instability, and anti-disturbance.

[0008] Furthermore, meteorological data settings are used to simulate different weather conditions. Meteorological data are configured through configuration files. After the configuration is completed, they are input into the weather simulator to generate simulated meteorological conditions. The meteorological data include light, precipitation and visibility; sensor simulation is used to simulate the performance changes of smart device sensors under different weather conditions; visual effect simulation is used to simulate the visual effects of the scenes where smart devices perform tasks under various weather conditions; weather dynamics are used to simulate the changes in meteorological data over time; and task scenarios are used to simulate the weather conditions required for task requirements.

[0009] Furthermore, the working condition configuration includes a normal area section without static or dynamic interferences, which is used to test the basic performance of the smart device under standard road conditions; a section containing static interferences, which is used to test the obstacle avoidance and collision prevention capabilities of the smart device; a section containing dynamic interferences, which is used to test the traffic perception and traffic avoidance capabilities of the smart device; a section containing pedestrian interference items, which is used to test the performance of the smart device in cities or densely populated areas, where the percentage of pedestrians can be adjusted; a section containing false targets, which is used to test the performance of the smart device in an environment with interference, where the number and position of the false targets can be adjusted; an occlusion section, which simulates the situation where the line of sight is blocked, and is used to test the performance of the smart device under the condition of line of sight obstruction; an obstacle avoidance section simulates the situation where obstacle avoidance operations are required, and is used to test the obstacle avoidance and safety performance of the smart device; a speed adjustment section simulates different speed limits or speed changes, and is used to test the performance of the smart device under different speed conditions.

[0010] Furthermore, noise design: by adding noise that conforms to the Poisson distribution to the image to simulate the noise of electromagnetic interference; color distortion: the image is saved in the HSV format, and the values ​​of the H and S channels are directly modified to simulate the image color distortion caused by electromagnetic interference; image jitter: the image jitter is simulated by translating and rotating the image; image blur: Gaussian blur is used to simulate the image blur caused by electromagnetic interference; brightness instability: after converting the image to YUV format, the Y channel value is changed as a whole to simulate the instability of the image brightness; adversarial perturbation: the fast gradient sign method FGSM is used to simulate the adversarial perturbation of the image.

[0011] Furthermore, the index system evaluates the environmental adaptability of intelligent devices from four aspects: situation awareness, situation understanding, intelligent decision-making, and task execution. Among them, situation awareness is used to evaluate the information perception ability of intelligent devices in complex environments, situation understanding is used to evaluate the ability of intelligent devices to understand perceived information, intelligent decision-making is used to evaluate the ability of intelligent devices to make decisions after understanding environmental information, and task execution is used to evaluate the execution ability of intelligent devices after making decisions.

[0012] Furthermore, situational awareness includes six indicators, namely, the mean of perceived precision, the variance of perceived precision, the variance of recall, the recall rate, the intersection over union (IOU), and the F1 score; situational understanding includes the re-identification evaluation index ; Intelligent decision-making includes key decisions and decision accuracy indicators, and task execution includes execution progress and execution evaluation indicators.

[0013] Furthermore, the perception accuracy is Score, The score is used to measure the stability and correctness of smart devices in completing tasks in different scenarios. The calculation formula is: , where correctness is the weighted average of the precision rate obtained by smart devices in complex environments, and its stability It is the weighted average of the variance of the precision rate obtained by smart devices in complex environments; the recall rate is used to predict the probability that the smart device completes the task correctly, and the recall rate is the number of times the smart device completes the task correctly divided by the total number of completed tasks; the intersection over union (IOU) is used to measure the accuracy of the smart device in detecting the corresponding object in a specific data set; the F1 score is used to measure the performance of the smart device in binary classification of the target.

[0014] Re-ID evaluation index Used to measure the recognition of smart devices in complex environments and re-identification Performance, Re-ID Evaluation Metrics It is equal to the weighted average of the recognition ability score of the smart device in a complex environment and the re-recognition ability score of the smart device in a complex environment. During the task execution process, the recognition ability of the smart device is measured based on the time it takes for the smart device to correctly identify the task target for the first time. The re-identification capability is measured based on the time it takes for the smart device to complete the re-identification operation. ,make , then the score calculation method for the smart device recognition ability and re-recognition ability is: .

[0015] The decision-making accuracy of smart devices is equal to the correct rate of decisions made by smart devices in different scenarios; key decisions refer to the number of times smart devices make accurate decisions in different scenarios.

[0016] Furthermore, all weights are configured through configuration files.

[0017] Furthermore, the task that smart devices need to accomplish is target identification and tracking.

[0018] The beneficial effects that can be achieved by adopting the present invention include: 1. The present invention adopts a method for evaluating the environmental adaptability of intelligent devices by combining a virtual simulation environment with index calculation, which solves the problem of insufficient environmental adaptability testing of intelligent devices in performing tasks in different scenarios; 2. The present invention provides a basic test environment and test data support for the environmental adaptability evaluation process of intelligent devices; 3. The diversity of virtual simulation environments simulated by the present invention can more fully test the environmental adaptability of intelligent devices; 4. The present invention supports the customized and flexible setting of parameters such as real environment terrain, dynamic / static obstacles, false targets, signal noise, etc.; 5. The weather design of the virtual simulation environment is an important issue in environmental adaptability testing, which is of great significance for quantitatively describing the adaptability of intelligent devices in the device to specific weather conditions. The verification environment can not only provide virtual environments under various weather conditions, but also reduce testing costs, reduce dependence on the natural environment, and simulate various extreme weather conditions. 6. In order to simulate different road conditions in the verification environment, the present invention sets 8 different types of road sections, thereby helping to evaluate the performance of equipment in various scenarios and improve its adaptability and robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flowchart of the implementation steps of the method for evaluating the environmental adaptability of an intelligent device based on a simulation environment in an embodiment; Figure 2 is the index system described in the present invention; Figure 3 It is a schematic diagram of virtual simulation environment parameter configuration; Figure 4 (a) is a schematic diagram of the topography of environment No. 0; Figure 4 (b) is a schematic diagram of environmental topography No. 1; Figure 4 (c) is a schematic diagram of the topography of environment No. 2; Figure 4 (d) is a topographical diagram of environment No. 3; Figure 4 (e) is a topographical diagram of environment No. 4; Figure 4 (f) is a topographical diagram of environment No. 5; Figure 4 (g) is a topographical diagram of environment No. 6; Figure 4 (h) is a topographical diagram of environment No. 7; Figure 5 It is a schematic diagram of the environmental adaptability assessment results of smart devices; Figure 6 It is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0020] like Figure 6 As shown, the present invention is an environmental adaptability evaluation method for intelligent devices based on a simulation environment, including generating different scene data from three dimensions, namely, weather configuration, working condition configuration, and interference configuration, in the simulation software, wherein the weather configuration is realized by six aspects, namely, meteorological data, sensor simulation, visual effect simulation, weather dynamics, and task scene integration; the working condition configuration is realized by a random combination of eight different road sections to realize different road conditions; the interference configuration is realized by six aspects, namely, noise design, color distortion, image jitter, image blur, brightness instability, and anti-disturbance; the simulation software generates different scene data streams through a series of configurations and transmits them to the evaluation model, and then calculates the indicators in the indicator system according to a proposed set of indicator system evaluation models, and finally combines a comprehensive analysis method to evaluate the environmental adaptability of the intelligent device.

[0021] (1) Generate different scene data through different configurations and output them as scene data streams.

[0022] The simulation software generates data for different scenarios by configuring weather, road conditions, and interference. The following describes in detail how to configure weather, road conditions, and interference.

[0023] 1) Weather configuration: The weather design of the virtual simulation environment is an important issue in environmental adaptability testing. It is of great significance for quantitatively describing the adaptability of the intelligent algorithm in the device to specific weather conditions. The verification environment can not only provide a virtual environment under various weather conditions, but also reduce the test cost, reduce the dependence on the natural environment, and simulate various extreme weather conditions. The weather configuration is realized from the aspects of meteorological data, visual effect simulation, weather dynamics, etc.

[0024] First of all, meteorological data setting is a key part of weather simulation. Accurate meteorological data is required to simulate different weather conditions, including information such as light, precipitation, and visibility. These data can be configured through configuration files. These data can be input into the weather simulator in the verification environment to generate simulated meteorological conditions.

[0025] Secondly, sensor simulation is an important part of weather design in virtual simulation environments. The sensor performance of the device may be affected differently under different weather conditions. For example, in rainy day simulation, the camera may be disturbed by raindrops, resulting in degraded image quality. Therefore, the virtual simulation environment needs to simulate the performance changes of these sensors for testing and training.

[0026] The simulation of visual effects is also an important aspect in the virtual simulation environment. The visual effects under various weather conditions should be able to be accurately simulated and presented in the virtual environment. For example, in a rainy day simulation, the virtual environment should be able to display the visual effects of raindrops to increase the realism of the simulation.

[0027] Weather design in a virtual simulation environment also needs to consider the dynamic nature of weather. Weather conditions should not be static, but should be able to change over time. For example, in a rainy day simulation, the intensity of rainfall may gradually increase or decrease. This can increase the complexity of testing and training, allowing equipment to adapt to changing weather conditions.

[0028] Finally, the weather design of the virtual simulation environment also needs to integrate mission scenarios. Different missions may require different weather conditions, so the virtual environment should be able to set weather conditions according to mission requirements to ensure the realism and practicality of testing and training.

[0029] In summary, the weather design of the virtual simulation environment is a complex and diverse process, which needs to consider multiple factors such as meteorological data generation, sensor simulation, visual effects, weather dynamics, and mission scenario integration. Through reasonable weather design, a diverse and realistic environment can be provided for the environmental adaptability testing of intelligent algorithms, thereby improving their performance and reliability and ensuring that they can successfully perform tasks under various weather conditions.

[0030] 2) Working condition configuration: In order to simulate different road conditions in the verification environment, the present invention sets 8 different types of road sections, such as Figure 4 As shown in (a)-(h), it helps to evaluate the performance of equipment in various scenarios and improve its adaptability and robustness. The following will explore eight different types of virtual road sections, including their characteristics and design reasons, and why they are so important in the simulation verification environment.

[0031] First, the normal area segments (0) represent general road conditions without static or dynamic interference. The road conditions on these segments are relatively flat and there are not many special cases. The normal area segments are designed to test the basic performance of the algorithm under standard road conditions, such as path planning, control, etc. This segment type helps to evaluate the road adaptability and stability control of the algorithm to ensure that they can operate safely on normal roads.

[0032] Second, the road segment with static distractors (1) contains stationary distractors, such as cars parked on the roadside or other obstacles. The distractors on the road segment do not move, but they may become obstacles during the algorithm's driving process. This type of road segment is used to test the algorithm's obstacle avoidance and collision prevention capabilities. The algorithm needs to be able to detect and avoid obstacles on the road to ensure driving safety.

[0033] The road segment with dynamic distractors (2) takes into account moving distractors, such as other vehicles. These dynamic distractors can simulate real-world traffic conditions, including vehicle lane changes, overtaking, and other behaviors. This type of road segment is used to test the algorithm's traffic perception and traffic avoidance capabilities. The algorithm needs to be able to identify the position and speed of other vehicles and make corresponding driving decisions to avoid collisions and maintain traffic flow.

[0034] On the other hand, the segment with pedestrian interference (3) introduces pedestrian interference to simulate pedestrians walking and crossing on the road. Unlike the previous two segments, the key here is the adjustable percentage of pedestrians. This segment type is used to test the performance of the algorithm in cities or densely populated areas. The algorithm needs to have advanced pedestrian detection and avoidance technology to ensure that no accidents occur during driving.

[0035] The false target segment (4) takes into account the presence of false targets, which are not real but may mislead the algorithm's perception system. The number and location of false targets may vary, increasing the difficulty of the perception task. This segment type is used to test the performance of the algorithm in an environment with interference. The algorithm needs to be able to accurately identify real targets and exclude false targets to maintain highly reliable perception.

[0036] The blocked section (5) simulates the situation where the line of sight is blocked, that is, the algorithm may encounter obstacles or obstructions during driving. The line of sight may be partially blocked by buildings, trees, or other obstacles. This section type is used to test the performance of the algorithm in the case of obstructed line of sight. The algorithm needs to have strong perception and obstacle avoidance capabilities to cope with the risks that may be caused by obstacles.

[0037] Obstacle Avoidance Section (6) simulates situations where obstacle avoidance is required. This section type has no adjustable parameters, but requires the algorithm to automatically avoid obstacles to maintain safety. This section type is used to test the obstacle avoidance and safety performance of the algorithm. The algorithm needs to be able to react quickly and avoid obstacles to ensure driving safety.

[0038] Finally, the Speed ​​Adjustment segment (7) allows the speed to be adjusted, with the speed being a floating point number between 0.0 and 100.0 in kilometers per hour (km / h). By adjusting the speed, different speed limits or speed changes can be simulated. This segment type is used to test the performance of an algorithm under different speed conditions, where the algorithm needs to adapt to different speed requirements, such as high speeds on highways and low speeds on city roads.

[0039] In general, these eight different types of virtual road segments have their own unique characteristics and design reasons in the virtual simulation environment. Their existence enriches the testing, training, and development process, helps improve the performance and adaptability of the algorithm, and reduces the cost of testing and development. These virtual road segments provide important tools for various application fields, including autonomous vehicles, robotics, military simulation, urban planning, and traffic management, providing strong support for research and practice in different fields.

[0040] 3) Interference configuration: There are two main interference methods: electromagnetic interference and counter-interference. Simulated electromagnetic interference is mainly achieved through image noise, color distortion, unstable brightness, jitter, blur, etc. Since the scene data stream generated by the simulation software is in video format and each frame of the video is a picture, the implementation of the interference method is considered, mainly focusing on how to perform interference operations on pictures to achieve electromagnetic interference and counter-interference.

[0041] As for the noise problem, it is found through research that the characteristics of camera noise are consistent with Poisson distribution. Poisson noise appears as random point noise in the image, and does not form a uniform background in the image. This noise usually appears as discrete bright or dark spots, which is consistent with the noise characteristics caused by the randomness of the arrival of photons in the camera sensor. Therefore, modeling camera noise as Poisson distribution can better simulate the noise effect in actual conditions. Therefore, adding noise that conforms to Poisson distribution to the image simulates the noise of electromagnetic interference.

[0042] For the color distortion problem, we study the storage method of color information in the image. HSV (Hue, Saturation, Value) is a color model used to describe color, which is different from the common RGB (Red, Green, Blue) color model. The HSV model represents color in a more intuitive and intuitive way, allowing us to understand and manipulate color properties more easily. Therefore, the image is saved in HSV format, and the value of the H and S channels is directly modified to simulate the color distortion caused by electromagnetic interference.

[0043] For the brightness instability problem, we studied the brightness representation of the image. YUV is a color encoding format commonly used in digital video and image processing. Unlike the RGB (red, green, and blue) color model, YUV encodes color information separately from brightness information. This separation helps video compression and transmission while reducing the amount of data stored and transmitted. Therefore, after the image is converted to the YUV format, the brightness instability problem is simulated by changing the Y channel value as a whole.

[0044] The jitter of the image is simulated by translating and rotating the image. The function functions in the OpenCV function library are used to translate and rotate the image. To translate, we move each pixel of the image a certain distance in the horizontal and vertical directions. The specific process is completed by traversing each pixel of the image and copying it to the new position. When calculating the pixel position after translation, we use the bilinear interpolation method to process the pixel values ​​to ensure that the translated image looks smooth. Rotating an image usually involves rotating the image around its center point by a certain angle. We use affine transformation to complete it. The rotated image also uses the bilinear interpolation method to calculate the pixel values ​​to ensure that the rotated image looks smooth. When translating and rotating, it is necessary to handle the situation of the image boundary to ensure that the image does not exceed the boundary or leave blank areas. We cropped the pixels that exceeded the image resolution and filled the boundary with black pixels.

[0045] For image blurring, we use Gaussian blur in the OpenCV library to simulate image blurring caused by electromagnetic interference. The size of the blur kernel is set according to the degree of blurring. Generally, the larger the blur kernel size, the blurrier it is. The Gaussian blur kernel is a two-dimensional weight matrix, where the weight at the center point is the highest and the weights around it gradually decrease. The distribution of weights follows the shape of a Gaussian distribution, hence the name Gaussian blur. We calculate the weight values ​​at each position based on the size of the set Gaussian blur kernel. The core operation of Gaussian blur is to convolve the image with the generated Gaussian blur kernel. The convolution operation multiplies the value of each pixel by the weight of the kernel and sums them, and then assigns the result to the new value of the pixel. This operation is performed for each pixel in the image, thereby achieving a blurring effect for the entire image. When performing the convolution operation, it is necessary to handle the image boundary situation to ensure that the blurring effect can also be applied correctly at the boundary. We use the zero padding method, that is, adding zero pixels outside the image boundary and then performing the convolution operation.

[0046] For the problem of adversarial perturbation, the Fast Gradient Sign Method (FGSM) is used to simulate adversarial perturbations of images. The core idea of ​​FGSM is to add perturbations in the gradient direction of the input data so that the model produces incorrect classification results. FGSM perturbations are calculated according to the following formula: ,in is the generated adversarial image, is the original picture, is a hyperparameter that controls the size of the perturbation, represents the gradient calculation, represents the loss function, is the true label of the original input image.

[0047] Through the above methods, we successfully simulated different degrees of visual problems such as noise, color distortion, unstable brightness, jitter, blur, etc. under electromagnetic interference.

[0048] Through configurations such as weather, road conditions, and interference, the simulation software can generate different scenario data, providing a basis for subsequent evaluation model calculations.

[0049] (ii) Index system, used to evaluate the performance of smart devices. The input of this module is the scene data generated by the simulation software. The data stream is in video format and output to the evaluation model through NDI and UDP protocols as input. Then the evaluation model calculates the index score based on the index system. The index system proposed in this invention is used to evaluate the environmental adaptability of smart devices in different scenarios. Figure 2 As shown in Figure 1, the index system evaluates the environmental adaptability of intelligent devices from four aspects: situation awareness, situation understanding, intelligent decision-making, and task execution. Situation awareness includes six indicators, namely, the mean of perception accuracy, the variance of perception accuracy, the variance of recall rate, the recall rate, the IOU intersection-over-union ratio, and the F1 score. Among them, the perception accuracy is Score; Situational understanding including re-identification evaluation indicators ; Intelligent decision-making includes key decisions and decision accuracy indicators, and task execution includes execution progress and execution evaluation indicators.

[0050] During the indicator design process, existing algorithm evaluation indicators were referenced.

[0051] 1) Mainly used to measure the stability of the algorithm in a complex environment ( ) and correctness ( ) performance. Among them, the algorithm correctness is the weighted average of the precision rate obtained by the algorithm in a complex environment; the algorithm stability is the weighted average of the variance of the precision rate obtained by the algorithm in a complex environment. The weights here will be emphasized according to the specific environment and task characteristics. Let TP be the statistical count of the target correctly identified in the test, and FP be the statistical count of the misidentified, then the precision rate of the target recognition algorithm ( ) is calculated as: , put the algorithm to be tested into the test environment described in the test environment setting and complete the test according to the test process description, set an unmanned vehicle city target tracking explanation Each parameter is shown in Table 1.

[0052] Table 1 Data record of unmanned vehicle urban target tracking test

[0053] According to the test results in the unmanned vehicle urban target tracking test data record in Table 1, the environmental adaptability score calculation process is as follows: The score mainly considers the correctness of the algorithm in a complex environment and algorithm stability , and its calculation formula is Considering that the proportion of sunny days in a city is higher in a year, the weights of weather are set to 50% for sunny days, 20% for rainy days, 15% for foggy days and 15% for snowy days. and algorithm stability The calculation formula is , , therefore, the variance and mean of the perceptual accuracy are The mean and variance of the scores.

[0054] 2) Re-identification evaluation: The recognition and re-identification ability scores are calculated by and{ }Calculated, considering that there are many influencing factors in the city, the possibility of re-identification problem is very high, so the weight of re-identification score is higher, and its calculation formula is: , the re-identification evaluation is value.

[0055] 3) Recall rate: The recall rate is used to predict the ratio of the number of correctly identified targets to the total number of identified targets by the smart device. The larger the ratio, the better the performance of the smart device. The calculation formula is: , where recall represents the recall rate, TP represents the number of targets recognized correctly by the smart device, and FN represents the number of targets recognized incorrectly by the smart device.

[0056] 4) IOU: IOU is a measure of the accuracy of detecting corresponding objects in a specific data set. IOU represents the overlap rate between the generated candidate bound and the original marked bound, that is, the ratio of their intersection to their union. The higher the value, the higher the correlation, which is expressed as , where IOU stands for intersection over union, It indicates the candidate box that the smart device identifies the target. Represents the ground-truth box of the object to be recognized.

[0057] 5) F1 score: F1 score is used to measure the performance of smart devices in binary classification of targets. It takes into account the precision and recall of the model. The calculation formula of F1 score is: , the F1 score ranges between 0 and 1, where 1 represents a perfect classifier and 0 represents the worst classifier. The F1 score is particularly useful in cases where there is an imbalanced class because it takes into account the number of objects that the smart device identifies as correct and incorrect, rather than just the accuracy.

[0058] 6) Task score: Mainly used to measure the recognition of algorithms in complex environments ( ) and re-identification ( ) performance. During the task execution, the recognition ability of the algorithm is measured based on the time it takes for the algorithm to correctly recognize the task target for the first time, and the re-recognition ability is measured based on the time it takes for the algorithm to complete the re-recognition operation. Let the consumption time be , then the calculation method of recognition ability and re-recognition ability is , where 10s is the upper limit of the time. If the recognition time exceeds 10s, the recognition or re-recognition failure is recorded and the score is 0. The execution progress is the Task score.

[0059] 7) Decision accuracy: The decision accuracy index of smart devices refers to the ability of smart devices to accurately make correct decisions when processing information and making decisions. Smart devices are usually equipped with various sensors and algorithms to perceive, analyze and judge environmental information, so as to make correct decisions. It can be measured by the decision accuracy rate, which is the proportion of correct decisions that smart devices can make when processing information and making decisions. The higher the decision accuracy rate, the stronger the decision-making ability of the smart device. The calculation method is: collect decision data of smart devices in different scenarios and record each decision result (correct or wrong). Count the number of correct decisions and the total number of decisions. Use the following formula to calculate the decision accuracy rate: ,in, is the number of correct decisions, is the total number of decisions, It is the decision accuracy rate.

[0060] 8) Critical decision: Critical decision refers to the number of times a smart device makes accurate decisions in different scenarios. When an accurate decision is made, it is defined as 0, otherwise it is 1.

[0061] The execution progress and execution evaluation indicators included in task execution are all existing methods and will not be repeated here.

[0062] All the weight values ​​mentioned above can be configured through the configuration file to ensure that the most suitable evaluation indicators are selected for different intelligent algorithm working environments.

[0063] like Figure 1 As shown, the specific implementation process of the present invention is as follows: (1) Run the configuration file and configure the scene parameters, such as Figure 3As shown. roadList is an integer array of arbitrary length, and the elements are integers between 0 and 6, representing the 7 different road sections mentioned above. All road sections set in roadList constitute the test environment during the entire test process. roadParamList is an array of the same length as roadList, and each element is the value of the adjustable parameter in the corresponding road section in the roadList array. roadWeight is an array of floating-point numbers, corresponding to the proportion of each road section in roadList in the entire environmental adaptability test process. The sum of the elements in roadWeight is 1.0.

[0064] (2) Run the configuration file software and configure the data enhancement parameters. lightList is an array of floating point numbers, corresponding to the light intensity of each road section in roadList. The data range is 0.0-1.0, with 0.0 representing the lowest light intensity and 1.0 representing the highest light intensity. SnowList is an array of floating point numbers, corresponding to the snowfall level of each road section in roadList. The data range is 0.0-1.0, with 0.0 representing the lowest snowfall and 1.0 representing the highest snowfall. RainList is an array of floating point numbers, corresponding to the rainfall level of each road section in roadList. The data range is 0.0-1.0, with 0.0 representing the lowest rainfall and 1.0 representing the highest rainfall. FogList is an array of floating point numbers, corresponding to the fog concentration of each road section in roadList. The data range is 0.0-1.0, with 0.0 representing the lowest fog intensity and 1.0 representing the highest fog intensity.

[0065] (3) Run the configuration file software and configure the data enhancement parameters. flipList is an array of floating-point numbers, which performs a flip operation on the image of each road section in roadList. 0 represents no flipping, 1 represents horizontal flipping, 2 represents vertical flipping, and 3 represents horizontal + vertical flipping. rotaList is an array of floating-point numbers, which performs a rotation operation on the image of each road section in roadList. The value range of the element is 0-360, and the unit is degree. gussList is an array of floating-point numbers, which performs a Gaussian blur operation on the image of each road section in roadList. The elements are floating-point numbers between 0.0 and 1.0. poisList is an array of floating-point numbers, which adds Poisson noise to the image of each road section in roadList. The elements are floating-point numbers between 0.0 and 1.0. coloList is an array of floating-point numbers, which adds color countermeasures to the image of each road section in roadList. The elements are: 0 (no processing) or floating-point numbers between 0.5 and 2.0. The elements are floating-point numbers between 0.0 and 1.0. attaList is an array of floating point numbers, corresponding to the image of each road section in roadList where the adversarial algorithm is added. The elements are 0 or 1, 1 means the adversarial algorithm is applied, and 0 means it is not applied.

[0066] (4) Run the configuration file software and configure the simulation software path.

[0067] (5) Save the configuration scenario parameters, data enhancement parameters, and simulation software path to a specific local folder.

[0068] (6) Start the intelligent device environment adaptability and credibility testing tool, and run the background and simulation software.

[0069] (7) When the program ends, the environmental adaptability score of the smart device is obtained, such as Figure 5 shown.

Claims

1. A method for evaluating the environmental adaptability of intelligent devices based on a simulation environment, characterized in that: By constructing a virtual simulation environment, different scenario data are generated from three dimensions: weather configuration, working condition configuration, and interference configuration. Then, based on the indicator system, the scores of the smart device in completing tasks in different scenarios are calculated, thereby completing the evaluation of the environmental adaptability of the smart device. The weather configuration is used to provide a virtual environment under various weather conditions, and the working condition configuration is used to provide simulated road conditions in a verification environment. The scenario data stream composed of the virtual environment under various weather conditions and the simulated road conditions in the verification environment is in video format, and each frame of the video is a picture. The interference configuration is used to simulate the disturbance caused by electromagnetic interference to the picture.

2. The method for evaluating the environmental adaptability of intelligent devices based on a simulation environment according to claim 1, characterized in that: The weather configuration is realized by six aspects: meteorological data, sensor simulation, visual effects simulation, weather dynamics, and mission scenario integration. The working condition configuration is realized by random combination of different road sections to achieve different road conditions. The interference configuration is realized by six aspects: noise design, color distortion, image jitter, image blur, brightness instability, and anti-disturbance.

3. The method for evaluating the environmental adaptability of intelligent devices based on a simulation environment according to claim 2, characterized in that: Meteorological data settings are used to simulate different weather conditions. Meteorological data is configured through configuration files. After configuration, it is input into the weather simulator to generate simulated meteorological conditions. Meteorological data includes light, precipitation and visibility. Sensor simulation is used to simulate the performance changes of smart device sensors under different weather conditions. Visual effect simulation is used to simulate the visual effects of the scenes where smart devices perform tasks under various weather conditions. Weather dynamics is used to simulate the changes in meteorological data over time. Task scenarios are used to simulate the weather conditions required for task requirements.

4. The method for evaluating the environmental adaptability of intelligent devices based on a simulation environment according to claim 2, characterized in that: The working condition configuration includes normal area sections without static or dynamic interferences, which are used to test the basic performance of smart devices under standard road conditions; sections with static interferences, which are used to test the obstacle avoidance and collision prevention capabilities of smart devices; sections with dynamic interferences, which are used to test the traffic perception and traffic avoidance capabilities of smart devices; sections with pedestrian interference items, which are used to test the performance of smart devices in cities or densely populated areas, where the percentage of pedestrians can be adjusted; sections with false targets, which are used to test the performance of smart devices in interference environments, where the number and position of false targets can be adjusted; occlusion sections, which simulate the situation where the line of sight is blocked, are used to test the performance of smart devices in the case of line of sight obstacles; obstacle avoidance sections simulate the situation where obstacle avoidance operations are required, which are used to test the obstacle avoidance and safety performance of smart devices; vehicle speed adjustment sections simulate different speed limits or speed changes, which are used to test the performance of smart devices under different speed conditions.

5. The method for evaluating the environmental adaptability of intelligent devices based on a simulation environment according to claim 2, characterized in that: Noise design: Simulate the noise of electromagnetic interference by adding noise that conforms to Poisson distribution to the image; Color distortion: The image is saved in HSV format, and the values ​​of H and S channels are directly modified to simulate the image color distortion caused by electromagnetic interference; Image jitter: Simulate image jitter by translating and rotating the image; Image blur: Simulate image blur caused by electromagnetic interference through Gaussian blur; Brightness instability: After converting the image to YUV format, change the Y channel value as a whole to simulate the instability of image brightness; Adversarial perturbation: Simulate the adversarial perturbation of the image through the fast gradient sign method FGSM.

6. The method for evaluating the environmental adaptability of intelligent devices based on a simulation environment according to claim 1, characterized in that: The index system evaluates the environmental adaptability of intelligent devices from four aspects: situation awareness, situation understanding, intelligent decision-making, and task execution. Among them, situation awareness is used to evaluate the information perception ability of intelligent devices in complex environments, situation understanding is used to evaluate the ability of intelligent devices to understand the perceived information, intelligent decision-making is used to evaluate the ability of intelligent devices to make decisions after understanding environmental information, and task execution is used to evaluate the execution ability of intelligent devices after making decisions.

7. The method for evaluating the environmental adaptability of intelligent devices based on a simulation environment according to claim 6, characterized in that: Situational awareness includes six indicators, namely, mean perception accuracy, variance of perception accuracy, variance of recall, recall rate, intersection over union (IOU), and F1 score; situational understanding includes re-identification evaluation indicators ; Intelligent decision-making includes key decisions and decision accuracy indicators, and task execution includes execution progress and execution evaluation indicators.

8. The method for evaluating the environmental adaptability of intelligent devices based on a simulation environment according to claim 7, characterized in that: The perceived accuracy is Score, The score is used to measure the stability and correctness of smart devices in completing tasks in different scenarios. The calculation formula is: , where correctness is the weighted average of the precision rate obtained by smart devices in complex environments, and its stability It is the weighted average of the variance of the precision rate obtained by smart devices in complex environments; the recall rate is used to predict the probability of a smart device completing a task correctly. The recall rate is the number of times a smart device completes a task correctly divided by the total number of completed tasks; the intersection over union (IOU) is used to measure the accuracy of smart devices detecting corresponding objects in a specific data set; the F1 score is used to measure the performance of smart devices in binary classification of targets; Re-ID evaluation index Used to measure the recognition of smart devices in complex environments and re-identification Performance, Re-ID Evaluation Metrics It is equal to the weighted average of the recognition ability score of the smart device in a complex environment and the re-recognition ability score of the smart device in a complex environment. During the task execution process, the recognition ability of the smart device is measured based on the time it takes for the smart device to correctly identify the task target for the first time. The re-identification capability is measured based on the time it takes for the smart device to complete the re-identification operation. ,make , then the score calculation method for the smart device recognition ability and re-recognition ability is: ; The decision-making accuracy of smart devices is equal to the correct rate of decisions made by smart devices in different scenarios; key decisions refer to the number of times smart devices make accurate decisions in different scenarios.

9. A method for evaluating environmental adaptability of intelligent devices based on a simulation environment according to any one of claims 1 to 8, characterized in that: All weights are configured through the configuration file.

10. The method for evaluating the environmental adaptability of intelligent devices based on a simulation environment according to claim 9, characterized in that: The task that smart devices need to accomplish is target identification and tracking.

Images