A test method, a computer program product and a computer readable storage medium

Abstract

This specification discloses a test method, a computer program product, and a computer-readable storage medium. Test cases and corresponding simulated vehicle point clouds are generated according to the system parameters of each handling equipment and each simulated vehicle model file. Since the simulated vehicle models in each simulated vehicle model file are not completely the same and the parameters of each vehicle are different, simulated vehicle point clouds corresponding to simulated vehicle models of different shapes and sizes can be obtained, thereby testing the perception algorithm to be tested with simulated vehicle models of various shapes and sizes, and effectively verifying the recognition accuracy of the perception algorithm to be tested on vehicles of different shapes and sizes. In addition, the perception algorithm to be tested is simulated by the simulated vehicle point cloud corresponding to the target test case to obtain the simulation test result. No manual operation is required throughout the test, which greatly reduces the consumption of labor and time, improves the test efficiency, and reduces the test cost.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a testing method, a computer program product, and a computer-readable storage medium. Background Art

[0002] In modern industrial production and logistics management, handling equipment (such as forklifts) is widely used in warehouses, factories, and distribution centers. In order to improve work efficiency and safety, handling equipment can be equipped with a perception system to collect environmental information about the environment in which the handling equipment is located, and to assist the handling equipment in automatic identification and positioning. The perception system can be implemented through the cooperation of sensors and perception algorithms. For example, in the positioning and handling tasks of carriers such as pallets and cages, the radar sensor equipped on the handling equipment can generate point cloud data containing carrier information. The perception algorithm can identify the carrier from the point cloud data generated by the radar sensor and determine the position of the carrier relative to the handling equipment. It can be seen that ensuring the recognition accuracy of the perception algorithm can improve the accuracy of the handling equipment's perception system in perceiving the environment in which the handling equipment is located, thereby ensuring the efficiency and safety of the handling equipment in performing tasks. Based on this, how to ensure the recognition accuracy of the perception algorithm is one of the problems that need to be solved urgently.

[0003] In the prior art, a test scene can be manually constructed, and a handling device equipped with a radar sensor is placed in the test scene. A vehicle is deployed on the handling device, and the posture of the vehicle relative to the handling device is manually measured as the actual posture. After that, the point cloud data of the vehicle in the actual posture is obtained through the radar sensor as a test point cloud. In this way, the test point cloud and the actual posture of the vehicle are used to test whether the perception algorithm can correctly identify the posture of the vehicle.

[0004] However, in actual applications, perception algorithms may be required to identify and locate different vehicles, and different vehicles have different shapes and sizes. The above test scheme can only generate point cloud data based on vehicles with limited shapes and sizes, which greatly limits the diversity of test point clouds used when testing perception algorithms. In addition, the scheme of manually measuring the actual position of the vehicle is extremely labor-intensive and time-consuming, and has limited accuracy, which reduces the efficiency of testing perception algorithms. Summary of the invention

[0005] This specification provides a testing method, a computer program product, and a computer-readable storage medium to partially solve the above-mentioned problems existing in the prior art.

[0006] This manual adopts the following technical solutions:

[0007] This manual provides a test method, including:

[0008] Pre-acquire system parameters of each handling device and each simulation vehicle model file; the simulation vehicle model file includes a simulation vehicle model and various vehicle parameters, the simulation vehicle models included in different simulation vehicle model files are not completely the same, and the various vehicle parameters included in different simulation vehicle model files are different;

[0009] Determine each test case according to each handling equipment system parameter and each simulation vehicle model file;

[0010] Generate a simulation vehicle point cloud corresponding to each of the test cases;

[0011] In response to a simulation test request, a target test case among the test cases is called, and a simulation test is performed on the perception algorithm to be tested according to the simulated vehicle point cloud corresponding to the target test case to obtain a simulation test result of the perception algorithm to be tested.

[0012] Optionally, the system parameters of each handling equipment include at least one of the parameters of each handling equipment, the parameters of each sensor, the parameters of each simulation scene and the parameters of each posture transformation strategy; wherein, the simulated sensor described by the sensor parameters is configured on the simulated handling equipment model described by the handling equipment parameters, and the posture transformation strategy is used to indicate the transformation of the posture of the simulated handling equipment model or the posture of the simulated vehicle model.

[0013] Optionally, determining each test case according to each handling equipment system parameter and each simulation vehicle model file specifically includes:

[0014] Selecting at least one simulation vehicle model file from the simulation vehicle model files;

[0015] Selecting at least one group of handling equipment system parameters from the handling equipment system parameters; wherein the selected group of handling equipment system parameters includes at least one group of handling equipment parameters and at least one group of sensor parameters;

[0016] Generate test tasks according to the selected simulation vehicle model file and the selected handling equipment system parameters;

[0017] The simulation vehicle model file and the handling equipment system parameters included in the test task are arranged and combined to obtain test cases corresponding to the test task.

[0018] Optionally, determining each test case according to each handling equipment system parameter and each simulation vehicle model file specifically includes:

[0019] In response to the user's selection operation, at least one selected simulation vehicle model file is determined from the simulation vehicle model files, and at least one group of selected handling equipment system parameters is determined from the handling equipment system parameters; wherein the group of selected handling equipment system parameters includes at least one group of handling equipment parameters and at least one group of sensor parameters;

[0020] Generate test cases based on the selected simulation vehicle model file and the selected handling equipment system parameters.

[0021] Optionally, a set of selected handling equipment system parameters also includes at least one set of simulation scene parameters and / or at least one set of posture transformation strategies.

[0022] Optionally, the handling equipment system parameters in the test case include handling equipment parameters and sensor parameters;

[0023] Generating the simulated vehicle point clouds corresponding to the test cases, specifically including:

[0024] For each test case, according to the simulation vehicle model and various vehicle parameters contained in the simulation vehicle model file in the test case, determine the initial pose of the simulation vehicle model;

[0025] According to the handling equipment parameters in the test case, determine the initial posture of the simulation handling equipment model;

[0026] Determining a reference relative pose of the simulated carrier model relative to the simulated carrier model according to the initial pose of the simulated carrier model and the initial pose of the simulated carrier model;

[0027] According to the sensor parameters in the test case and the reference relative pose, a simulated vehicle point cloud corresponding to the test case is generated.

[0028] Optionally, the handling equipment system parameters in the test case also include a posture transformation strategy;

[0029] Determining a reference relative pose of the simulated carrier model relative to the simulated carrier model according to the initial pose of the simulated carrier model and the initial pose of the simulated carrier model specifically includes:

[0030] When the posture transformation strategy in the test case is the first posture transformation strategy, the initial posture of the simulated handling equipment model is transformed, and the reference relative posture of the simulated vehicle model relative to the simulated handling equipment model is determined according to the transformed posture of the simulated handling equipment model and the initial posture of the simulated vehicle model;

[0031] When the posture transformation strategy in the test case is the second posture transformation strategy, the initial posture of the simulated vehicle model is transformed, and the reference relative posture of the simulated vehicle model relative to the simulated handling equipment model is determined based on the initial posture of the simulated handling equipment model and the transformed posture of the simulated vehicle model.

[0032] Optionally, the handling equipment system parameters in the test case also include simulation scenario parameters;

[0033] Generating a simulated vehicle point cloud corresponding to the test case according to the sensor parameters in the test case and the reference relative pose, specifically including:

[0034] According to the sensor parameters and simulation scene parameters in the test case, as well as the reference relative posture, a simulation vehicle point cloud corresponding to the test case is generated.

[0035] Optionally, performing a simulation test on the perception algorithm to be tested according to the simulated vehicle point cloud corresponding to the target test case to obtain a simulation test result of the perception algorithm to be tested specifically includes:

[0036] The simulated vehicle point cloud corresponding to the target test case is input into the perception algorithm to be tested, and the target relative posture of the simulated vehicle model relative to the simulated handling equipment model in the target test case is identified from the simulated vehicle point cloud corresponding to the target test case through the perception algorithm to be tested;

[0037] The reference relative posture of the simulated vehicle model relative to the simulated handling equipment model in the target test case is used as a reference, and the simulation test result of the perception algorithm to be tested is determined according to the difference between the target relative posture and the reference relative posture.

[0038] Optionally, one test case corresponds to multiple groups of simulated vehicle point clouds, and different groups of simulated vehicle point clouds respectively correspond to different reference relative postures of the simulated vehicle models relative to the simulated handling equipment model.

[0039] Optionally, performing a simulation test on the perception algorithm to be tested according to the simulated vehicle point cloud corresponding to the target test case to obtain a simulation test result of the perception algorithm to be tested specifically includes:

[0040] For each group of simulated vehicle point clouds corresponding to the target test case, the group of simulated vehicle point clouds is input into the perception algorithm to be tested, and the target relative posture of the simulated vehicle model corresponding to the group of simulated vehicle point clouds relative to the simulated handling equipment model is identified from the group of simulated vehicle point clouds by the perception algorithm to be tested;

[0041] Taking the reference relative pose of the simulated vehicle model corresponding to the group of simulated vehicle point clouds relative to the simulated handling equipment model as a reference, and determining the simulation test result of the perception algorithm to be tested detecting the group of simulated vehicle point clouds according to the difference between the reference relative pose corresponding to the group of simulated vehicle point clouds and the target relative pose corresponding to the group of simulated vehicle point clouds;

[0042] The simulation test results of each group of simulated vehicle point clouds corresponding to the target test case are detected respectively according to the perception algorithm to be tested, and the simulation test results of the perception algorithm to be tested are determined.

[0043] Optionally, different groups of simulated vehicle point clouds correspond to different test distances, where the test distance is the distance between a vehicle fork entry surface of the simulated vehicle model and a vehicle body center of the simulated handling equipment model.

[0044] Optionally, the perception algorithm to be tested includes multiple perception algorithms, and different perception algorithms detect different features of the vehicle when identifying the vehicle posture.

[0045] Optionally, performing a simulation test on the perception algorithm to be tested according to the simulated vehicle point cloud corresponding to the target test case to obtain a simulation test result of the perception algorithm to be tested specifically includes:

[0046] For each perception algorithm to be tested, the simulated vehicle point cloud corresponding to the target test case is input into the perception algorithm to be tested, and the perception algorithm to be tested detects the characteristics of the vehicle and identifies the relative pose of the simulated vehicle model in the target test case relative to the simulated handling equipment model from the simulated vehicle point cloud corresponding to the target test case as the target relative pose output by the perception algorithm to be tested;

[0047] The reference relative posture of the simulated vehicle model relative to the simulated handling equipment model in the target test case is used as a reference, and the simulation test result of the perception algorithm to be tested is determined according to the difference between the target relative posture output by the perception algorithm to be tested and the reference relative posture.

[0048] Optionally, the method further comprises:

[0049] Pre-acquire a real vehicle point cloud corresponding to the target test case;

[0050] In response to a real test request, the perception algorithm to be tested is tested according to the real vehicle point cloud corresponding to the target test case to obtain a real test result of the perception algorithm to be tested;

[0051] A comprehensive test result of the perception algorithm to be tested is determined according to the simulation test result and the actual test result.

[0052] Optionally, obtaining each simulation vehicle model file in advance specifically includes:

[0053] Pre-receive each simulated vehicle model and the corresponding size of each simulated vehicle model input by the user;

[0054] Determining parameters of each vehicle according to the sizes corresponding to each simulation vehicle model;

[0055] Generate each simulation vehicle model file according to each simulation vehicle model and each vehicle parameter.

[0056] Optionally, the method further comprises:

[0057] Performing statistics on simulation test results of the perception algorithm to be tested to determine the accuracy of the perception algorithm to be tested in detecting the pose of the simulated vehicle model corresponding to the target test case;

[0058] The accuracy of the posture of the simulated vehicle model corresponding to the target test case is detected according to the determined perception algorithm to be tested, and a test report corresponding to the perception algorithm to be tested is generated.

[0059] Optionally, performing a simulation test on the perception algorithm to be tested according to the simulated vehicle point cloud corresponding to the target test case to obtain a simulation test result of the perception algorithm to be tested specifically includes:

[0060] Get the preset query cycle and the preset test timeout;

[0061] Whenever the query cycle is reached, query whether there is a simulation test result of the perception algorithm to be tested obtained by executing the target test case; if so, update the test status of the target test case to completed, and display the completed test status of the target test case to the user; if not, continue to wait;

[0062] When the preset test timeout is reached, and the simulation test result of the perception algorithm to be tested obtained by executing the target test case does not exist, the test status of the target test case is updated to abnormal, and the abnormal test status of the target test case is displayed to the user.

[0063] This specification provides a computer program product, including a computer program, which implements the above-mentioned testing method when executed by a processor.

[0064] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned test method is implemented.

[0065] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0066] In the test method provided in this specification, test cases and corresponding simulated vehicle point clouds are generated according to the system parameters of each handling equipment and each simulated vehicle model file. Since the simulated vehicle models in each simulated vehicle model file are not completely the same and the parameters of each vehicle are different, simulated vehicle point clouds corresponding to simulated vehicle models of different shapes and sizes can be obtained, so that the perception algorithm to be tested is tested with simulated vehicle models of various shapes and sizes, and the recognition accuracy of the perception algorithm to be tested on vehicles of different shapes and sizes is effectively verified. In addition, the simulation test results are obtained by simulating the perception algorithm to be tested with the simulated vehicle point cloud corresponding to the target test case. No manual operation is required throughout the test, which greatly reduces the consumption of labor and time, improves the test efficiency and reduces the test cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings:

[0068] Figure 1 A flow chart of a test method in this manual;

[0069] Figure 2 A schematic diagram of the architecture of a test platform in this specification;

[0070] Figure 3 A schematic diagram of a vehicle selection interface in this manual;

[0071] Figure 4 A schematic diagram of a handling device and sensor parameter selection interface in this manual;

[0072] Figure 5 A schematic diagram of a strategy and scenario selection interface in this manual;

[0073] Figure 6 A schematic diagram of a task generation interface in this manual;

[0074] Figure 7 A schematic diagram of a test case customization interface in this manual;

[0075] Figure 8 This is a schematic diagram of a simulation vehicle model file management interface in this manual;

[0076] Fig. 9 This is a schematic diagram of a handling equipment parameter management interface in this manual;

[0077] Fig.10This is a schematic diagram of a sensor parameter management interface in this manual;

[0078] Fig.11 This is a schematic diagram of a simulation scene parameter management interface in this manual. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0080] In addition, it should be noted that all actions of acquiring signals, information or data in the present invention are performed in compliance with the corresponding data protection laws and policies of the location and with the authorization given by the owner of the corresponding device.

[0081] In the absence of conflict, the features of the following embodiments and implementations may be combined with each other.

[0082] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0083] As mentioned above, in existing perception algorithm testing solutions, the test scene is generally constructed manually according to the actual scene where the perception algorithm is applied, which can be indoors or outdoors. A handling device is placed in the test scene, and the handling device is equipped with sensors and perception algorithms. After placing the vehicle in the test scene, the posture of the vehicle relative to the handling device can be manually measured as the actual posture. The sensors equipped on the handling device can obtain the point cloud data of the vehicle as the test point cloud, and the test point cloud is submitted to the perception algorithm for analysis. The perception algorithm can identify and locate the vehicle, obtain the estimated posture of the vehicle relative to the handling device, compare the estimated posture with the actual posture, and determine the accuracy of the perception algorithm positioning. However, the above solution has several defects:

[0084] First: The handling equipment and carriers in the above scheme are physical handling equipment and physical carriers. The shapes and sizes of physical handling equipment and physical carriers are limited and cannot be flexibly changed in the test scenario. In particular, there are various types, shapes and sizes of carriers. The test point cloud obtained in the test scenario is difficult to cover the combination of different types, shapes and sizes, resulting in the perception algorithm being able to only be tested based on the test point cloud generated by a limited number of carriers. In actual applications, the perception algorithm will face various types, shapes and sizes of carriers, which may cause the perception algorithm to meet the performance requirements during testing but perform poorly in actual applications. The perception algorithm has to be tested again based on more types, shapes and sizes of carriers, greatly reducing the efficiency of the test.

[0085] Second: In order to expand the test data, sensors are usually required to obtain point cloud data in different postures as test point clouds. Under this premise, it is necessary to manually move the vehicle multiple times and manually measure the vehicle posture multiple times, and the sensor scans in multiple times to obtain test point clouds in multiple actual postures; or manually place the vehicle, measure the vehicle posture, obtain the point cloud data of the vehicle in the actual posture, and then import it into the test platform, and change the posture corresponding to the point cloud data on the test platform multiple times to obtain test point clouds in multiple actual postures. Obviously, the solution of manually moving the vehicle or manually measuring the vehicle posture is extremely labor-intensive and time-consuming, and the accuracy of manually measuring the vehicle posture is limited, which also reduces the test efficiency of the perception algorithm.

[0086] Third: The perception algorithm actually recognizes and locates based on the test point cloud within a certain range of a target point. If the target point identified by the perception algorithm is far away from the test point cloud in the vehicle center coordinate system, the point cloud data used to estimate the pose is relatively sparse, and the perception algorithm may not be able to perform vehicle pose recognition normally, or even completely miss the existence of the vehicle. During the test, the actual poses of the vehicles corresponding to different groups of test point clouds are different. When the test point cloud is replaced, the target point used to test the perception algorithm based on the previous group of test point clouds may not be suitable for the current group of test point clouds. The above-mentioned problem of the target point being far away from the test point cloud is likely to occur. Therefore, every time the test point cloud is replaced, the tester needs to manually modify the target point on the test platform to ensure that the perception algorithm can normally recognize the current group of test point clouds. This process is extremely dependent on the experience of the tester and is extremely time-consuming, resulting in a decrease in test efficiency.

[0087] Based on this, this specification provides a test method, which generates a simulated vehicle point cloud through the pre-acquired system parameters of each handling equipment and each simulated vehicle model file, and uses the simulated vehicle point cloud to simulate the perception algorithm to be tested. Since the simulated vehicle models contained in different simulated vehicle model files are not exactly the same, the simulated point cloud generated based on simulated vehicle models of different types, shapes and sizes can be tested, so that the simulation test can cover more actual situations. In addition, there is no need to manually move the vehicle, manually measure the vehicle posture, or manually modify the target point during the entire test, which saves labor costs and time costs while improving the accuracy of the actual posture, thereby improving the test efficiency.

[0088] Figure 1 The following is a flow chart of a test method in this specification, which specifically includes the following steps:

[0089] S100: pre-acquire system parameters of each transport device and each simulation vehicle model file. The simulation vehicle model file includes a simulation vehicle model and various vehicle parameters. The simulation vehicle models included in different simulation vehicle model files are not completely the same, and the various vehicle parameters included in different simulation vehicle model files are different.

[0090] A test method provided in an embodiment of this specification can be executed by an electronic device for testing. The electronic device can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services, which is not limited in this specification. For the convenience of explanation, in one or more subsequent embodiments of this specification, Figure 2The test platform shown in the figure executes the test method provided in this specification as an example, and the specific embodiment is described in detail. The test platform 200 includes at least a front-end interactive system 210 and a back-end management system 220, wherein the front-end interactive system 210 is used to display various interfaces to the user through a terminal device 230, and receive information (such as handling equipment parameters, sensor parameters, simulation vehicle models, vehicle parameters, simulation scene parameters, etc.) and operations (such as generating test tasks and test cases, executing simulation tests, outputting test reports, etc.) input by the user in each interface through the terminal device 230, and forward the acquired information and operations to the server 240 of the back-end management system 220, and the server 240 of the back-end management system 220 can be used to receive information input by the user through the front-end interactive system 210, and store the received information in the database 250 through a specific interface, and execute the generation of test tasks, the generation of test cases, the testing of the perception algorithm to be tested, the generation of test reports, etc. in response to the user's operation. The front-end interactive system 210 and the back-end management system 220 can communicate through any existing type of communication media, such as a wired communication link or a wireless communication link, and the wireless communication link includes but is not limited to: Bluetooth, Wi-Fi, near field communication, cellular mobile communication network, to achieve the reception or transmission of information and operations. In addition, Figure 2 The front-end interactive system 210 shown includes only one terminal device 230, but according to the actual application scenario, multiple terminal devices 230 can be deployed in the front-end interactive system 210, and communication connections can be established with the server 240 of the back-end management system 220 respectively to provide testing functions for multiple users respectively.

[0091] Specifically, the test platform can pre-acquire the system parameters of each handling equipment and each simulation vehicle model file, and store them in the database. Among them, the simulation vehicle model file contains the simulation vehicle model and each vehicle parameter, and the simulation vehicle models contained in different simulation vehicle model files are not completely the same, and the vehicle parameters contained in different simulation vehicle model files are different. The handling equipment system parameters are used to describe other simulation models related to the test task in the simulation scene and the simulation scene itself, and may include one or more of the handling equipment parameters used to describe the simulation handling equipment model, the sensor parameters used to describe the simulation sensor, the posture transformation strategy used to determine the reference relative posture of the simulation vehicle model relative to the simulation handling equipment model, and the simulation scene parameters used to describe the simulation scene itself. The system parameters of each handling equipment pre-acquired by the test platform in this specification can be determined based on specific test requirements, and this specification does not specifically limit the parameter type and quantity contained in the pre-acquired system parameters of each handling equipment.

[0092] The simulation vehicle model file includes the simulation vehicle model and its corresponding vehicle parameters. The simulation vehicle model can be a three-dimensional model. The vehicle parameters may include vehicle type, vehicle entry surface size, entry point offset in all directions relative to the origin, vehicle material and color, pier width, pier height, hole width, panel height, floor height, panel protrusion, column width, column spacing, entry height offset relative to beam height, overall upper and lower offset distance of the area of ​​interest, and other data used to describe the simulation vehicle model.

[0093] For each simulation vehicle model file, the simulation vehicle models respectively included in different simulation vehicle model files may be the same or different, but generally the vehicle parameters respectively included in different simulation vehicle model files are different from each other. Therefore, even if the simulation vehicle models respectively included in two simulation vehicle model files are the same, due to the difference in each vehicle parameter, the two simulation vehicle model files may also respectively describe simulation vehicle models of different sizes. Since the simulation vehicle models in each simulation vehicle model file are not completely the same and the vehicle parameters are different, the simulation vehicle point clouds corresponding to simulation vehicle models of different types, shapes and sizes can be obtained, and simulation tests can be performed in this way, so that the simulation tests can cover more situations, without being restricted by the type, shape, and size of the physical vehicle, and without being restricted by whether the physical vehicle is in a real test site.

[0094] The above-mentioned simulation carrier model files and system parameters of each handling equipment can be configured by the user through Figure 2 The terminal device input in the front-end interactive system of the test platform shown is customized by the user; it can also be determined based on the product parameters of existing physical products such as physical carriers, physical handling equipment, physical sensors, etc. that are pre-entered into the back-end management system of the test platform and the actual working scenario, and this manual does not limit this.

[0095] In one or more embodiments of the present specification, each handling equipment system parameter may include at least each sensor parameter and each handling equipment parameter. The simulated sensor described by the sensor parameter is configured on the simulated handling equipment model described by the handling equipment parameter, and the specific configuration position may be the body part or the accessory part of the simulated handling equipment model. For a simulated handling equipment model, the number of simulated sensors configured thereon, the configuration posture, and the sensor parameters corresponding to each simulated sensor can be flexibly determined based on the specific test task (use case), and no specific limitation is made here.

[0096] In this specification, the simulated handling equipment model described by the handling equipment parameters may include any one of an automated guided vehicle (AGV), an autonomous mobile robot (AMR), a humanoid robot, a forklift, a gripper, a stacker, and a tractor. The handling equipment parameters may include the handling equipment identification, the handling equipment dimensions (width, length, height of the handling equipment), the attachment dimensions (such as fork length, fork width, fork thickness), the leg height, the mast posture of the handling equipment, and the attachment posture, etc., which are used to describe the data of the simulated handling equipment model. Based on the handling equipment parameters, a simulated handling equipment model with a specific shape and size can be constructed, and applied to the simulation test of the perception algorithm to be tested, so as to simulate the physical handling equipment in the real scene.

[0097] The sensor parameters may include the identification of the simulated sensor, the type of the simulated sensor, the external parameters of the simulated sensor (i.e., the configuration position of the simulated sensor on the simulated handling equipment model), and the internal parameters of the simulated sensor. The types of simulated sensors include radar sensors (such as laser radars, millimeter wave radars, etc.), stereo vision cameras, ultrasonic sensors, and any other existing type of sensors that can obtain point cloud data.

[0098] S102: Determine each test case according to the system parameters of each handling equipment and the model files of each simulation vehicle.

[0099] Furthermore, in order to perform simulation tests on the perception algorithm to be tested, a simulation vehicle model can be placed in the simulation coordinate system so as to generate a simulation vehicle point cloud containing at least the simulation vehicle model in the simulation coordinate system. The simulation coordinate system is a reference frame for defining and describing the position, posture, and motion of the simulation model (including at least the simulation vehicle model, the simulation handling equipment model, and the simulation sensor) in the computer simulation environment provided by the test platform. The type, shape, and size of the simulation vehicle model to be placed in the simulation coordinate system, as well as the simulation scene to be built, can be determined based on the test case. Based on this, each test case can be generated according to the system parameters of each handling equipment and each simulation vehicle model file pre-acquired in step S100. This specification does not limit the number of simulation vehicle model files and the number of handling equipment system parameters contained in each test case.

[0100] In one or more embodiments of the present specification, each test case includes at least one simulation vehicle model file and a set of handling equipment system parameters, wherein the set of handling equipment system parameters may include at least one set of handling equipment parameters and at least one set of sensor parameters. That is, when testing the perception algorithm to be tested based on one test case, at least one simulation handling equipment model and one simulation vehicle model are placed in the simulation coordinate system, and at least one simulation sensor is placed on the simulation handling equipment model. There is at least one difference in the simulation vehicle model files and handling equipment system parameters respectively included in different test cases, so that repeated testing of the perception algorithm to be tested based on the same simulation vehicle model, the same handling equipment system parameters and the same sensor parameters can be avoided in the simulation test, so as to improve the test efficiency.

[0101] In actual applications, due to the large number of simulation vehicle model files and handling equipment system parameters pre-acquired by the test platform, the scale of test cases with different contents generated based on all simulation vehicle model files and all handling equipment system parameters entered by the test platform is still very large. To this end, the user can perform the generation of test cases through the front-end interactive system of the test platform according to specific test requirements. Afterwards, at least one simulation vehicle model file is selected from each simulation vehicle model file, and at least one group of handling equipment system parameters is selected from each group of handling equipment system parameters. The front-end interactive system can generate test cases that meet the user's test requirements based on the user's selection. In addition, multiple test cases can also be generated based on a user's one selection. Specifically, the user can perform the generation of test tasks through the front-end interactive system of the test platform. In this way, the test platform can generate test tasks based on each simulation vehicle model file and each handling equipment system parameter selected by the user, and traverse the simulation vehicle model files and each handling equipment system parameter contained in the test task for permutation and combination, so as to obtain multiple test cases.

[0102] S104: Generate a simulation vehicle point cloud corresponding to each of the test cases.

[0103] The test method provided in this specification is to test the accuracy of the perception algorithm to be tested in identifying the position of the carrier relative to the handling equipment from the point cloud data. Therefore, a simulated vehicle point cloud corresponding to each test case can also be generated, so that the simulated vehicle point cloud corresponding to the test case can be used as a test point cloud to simulate the perception algorithm to be tested.

[0104] Specifically, for each test case, a simulated handling equipment model is constructed according to the handling equipment parameters in the handling equipment system parameters contained in the test case, and is placed in the simulation coordinate system. According to the sensor parameters in the handling equipment system parameters contained in the test case, the simulated sensor is deployed on the placed simulated handling equipment model. Afterwards, the simulated vehicle model is placed in the simulation coordinate system according to the simulated vehicle model file contained in the test case. It should be noted here that the aforementioned is only an optional placement order for each simulation model. This specification does not limit the placement order of the aforementioned simulation models. The simulated vehicle model can be placed on the simulated handling equipment model, or it can be placed near the simulated handling equipment model instead of on the simulated handling equipment model. This specification does not limit the placement relationship between the simulated vehicle model and the simulated handling equipment model.

[0105] In the simulation coordinate system, the pose of the simulation vehicle model can be the default initial pose, or the transformed pose after automatic transformation or manual adjustment by the user. Similarly, the pose of the simulation handling equipment model can also be the default initial pose or the transformed pose after automatic transformation or manual adjustment by the user. Then, according to the pose of the simulation handling equipment model and the pose of the simulation vehicle model, the reference relative pose of the simulation vehicle model relative to the simulation handling equipment model is determined. With the simulation sensor deployed on the simulation handling equipment model, under the reference relative pose of the simulation vehicle model relative to the simulation handling equipment model, the point cloud data generation simulation is started to simulate the process of the physical sensor scanning the physical vehicle to obtain the simulated vehicle point cloud. Compared with the current solution of obtaining the point cloud of the physical vehicle as the test point cloud through the physical sensor, since the simulated vehicle point cloud is automatically generated through simulation, the efficiency of obtaining the simulated vehicle point cloud has been greatly improved, and at the same time, the time cost and labor cost are reduced, thereby improving the test efficiency of the perception algorithm to be tested.

[0106] At the same time, since the reference relative posture of the simulated vehicle model relative to the simulated handling equipment model in this specification is automatically calculated in the simulation coordinate system, compared with the current testing process in which the actual posture of the physical vehicle relative to the physical handling equipment is manually measured, the reference relative posture obtained in the simulation coordinate system has higher accuracy and smaller error, thereby improving the reliability of the simulation test results of the perception algorithm to be tested and improving the test efficiency of the simulation test.

[0107] The generated simulated vehicle point cloud, test case, and the reference relative pose of the simulated vehicle model relative to the simulated handling equipment model when the simulated vehicle point cloud is generated are bound and stored in the database. Thus, when the test case is called to test the perception algorithm to be tested, the simulated vehicle point cloud corresponding to the test case can be queried as the test point cloud, and the reference relative pose of the corresponding simulated vehicle model relative to the simulated handling equipment model can be queried as a reference to evaluate the precision and accuracy of the recognition pose of the perception algorithm to be tested.

[0108] Optionally, if only the simulated vehicle model and the simulated handling equipment model are placed in the simulation coordinate system, the problem of the simulation environment being too idealized may occur, and then the perception algorithm to be tested may perform well in the idealized test environment, but may perform poorly when encountering complex situations in the actual working environment. In view of this situation, when generating the simulated vehicle point cloud, a simulation environment for simulating the actual working environment can be constructed according to the simulation scene parameters in the handling equipment system parameters. For example, in the simulation coordinate system, a number of simulated obstacles of different shapes, sizes, and positions are placed within a certain range near the simulated handling equipment model and the simulated vehicle model. In this way, the generated simulated vehicle point cloud will contain information about the simulation environment (such as the point cloud of some simulated obstacles), so that the perception algorithm to be tested needs to identify the simulated vehicle model and determine the pose of the simulated vehicle model from the simulated vehicle point cloud containing information about the simulation environment. Based on the simulation test results, the precision and accuracy of the perception algorithm to be tested in a complex environment can be verified to ensure that the perception algorithm to be tested can cope with various complex situations in the actual working environment.

[0109] S106: In response to the simulation test request, a target test case among the test cases is called, and a simulation test is performed on the perception algorithm to be tested according to the simulated vehicle point cloud corresponding to the target test case to obtain a simulation test result of the perception algorithm to be tested.

[0110] The user can select the perception algorithm to be tested and the target test case to be used in the test from the generated test cases through the front-end interactive system of the test platform. Of course, the test platform can also automatically select the target test case from the generated test cases based on the actual test situation. The front-end interactive system can generate a simulation test request based on the target test case and the perception algorithm to be tested selected by the user, and send the simulation test request to the back-end management system of the test platform. The back-end management system responds to the simulation test request, parses the perception algorithm to be tested and the target test case to be used in the test from the simulation test request, and can search the algorithm parameters of the perception algorithm to be tested and the information of the target test case from the database, so as to perform a simulation test on the perception algorithm to be tested according to the simulated vehicle point cloud corresponding to the target test case, and obtain the corresponding simulation test result.

[0111] The number of target test cases can be one or more. If multiple target test cases are used in one simulation test to simulate the perception algorithm to be tested, the simulated vehicle point cloud corresponding to each target test case can be used one by one to simulate the perception algorithm to be tested, and the simulation test results of the perception algorithm to be tested under each target test case can be obtained. By comprehensively analyzing the simulation test results of the perception algorithm to be tested under each target test case, quantitative indicators for characterizing the precision and accuracy of the perception algorithm to be tested can be obtained.

[0112] The perception algorithm to be tested can be any existing type of algorithm used to understand environmental information based on point cloud data obtained by sensors, such as a laser radar perception algorithm and a millimeter wave radar perception algorithm. In addition, different perception algorithms to be tested can also obtain the position of the vehicle relative to the handling equipment by detecting different characteristics of the vehicle, such as a pier hole detection algorithm and a socket detection algorithm. This specification does not specifically limit the type of perception algorithm to be tested actually used in the test, the characteristics of the detected vehicles, and the number.

[0113] In this specification, the perception algorithm to be tested can identify the vehicle from the point cloud data and determine the position and posture of the vehicle relative to the handling equipment, thereby completing the vehicle identification and positioning tasks, and then assisting the handling equipment to complete subsequent tasks such as moving, transporting, and stacking the vehicle.

[0114] In this specification, the test platform simulates the perception algorithm to be tested to identify the target relative posture of the simulated vehicle model relative to the simulated handling equipment model from the simulated vehicle point cloud corresponding to the target test case. In this process, the perception algorithm to be tested will identify the simulated vehicle point cloud within a certain range of the target point (for details, please refer to the third defect of the aforementioned existing solution). However, through the test method provided in this specification, even if multiple groups of simulated vehicle point clouds with different reference relative postures are used to simulate the perception algorithm to be tested, after the simulation test based on the previous group of simulated vehicle point clouds and before the simulation test based on the next group of simulated vehicle point clouds, the test platform can automatically change the target points used by the perception algorithm to be tested based on the next group of simulated vehicle point clouds. Without manual modification by the user, it can ensure that the perception algorithm to be tested can normally identify the next group of simulated vehicle point clouds, thereby improving the efficiency of the simulation test while lowering the threshold of the simulation test.

[0115] The simulation test results can characterize the precision and accuracy of the perception algorithm to be tested under the target test cases. Since the number of target test cases used for the simulation test of the perception algorithm to be tested can be multiple, the test method provided in this specification can be used to test the recognition precision and accuracy of the perception algorithm to be tested under the conditions of simulated vehicle models of different types, shapes and sizes, thereby avoiding the impact on the comprehensiveness of the test of the perception algorithm to be tested due to the limited types, shapes and sizes of physical vehicles.

[0116] After obtaining the simulation test results of the perception algorithm to be tested, the precision and accuracy of the perception algorithm to be tested can be analyzed based on the simulation test results to determine whether the perception algorithm to be tested needs to be adjusted and optimized. Of course, the perception algorithm to be tested can also be tested multiple times based on multiple different target test cases to iteratively adjust the perception algorithm to be tested until the precision and accuracy of the perception algorithm to be tested reaches the expected level.

[0117] based on Figure 1 The test method shown in the figure obtains the system parameters of each handling equipment and each simulation vehicle model file in advance. The simulation vehicle model file contains the simulation vehicle model and each vehicle parameter. Moreover, the simulation vehicle models contained in different simulation vehicle model files are not completely the same, and the parameters of each vehicle are different. According to each simulation vehicle model file and each handling equipment system parameter, each test case is determined, and the simulation vehicle point cloud corresponding to each test case is generated. When performing a simulation test, the perception algorithm to be tested is simulated and tested according to the simulation vehicle point cloud corresponding to the target test case to obtain a simulation test result.

[0118] It can be seen from the above method that this method can obtain the simulated vehicle point cloud corresponding to the simulated vehicle models of different shapes and sizes by constructing multiple simulated vehicle model files with different simulated vehicle models and different vehicle parameters, so as to test the perception algorithm to be tested with simulated vehicle models of various shapes and sizes, and effectively verify the recognition accuracy and accuracy of the perception algorithm to be tested on vehicles of different shapes and sizes. In addition, through the simulation test method, no manual operation is required throughout the test, which greatly reduces the waste of manpower and time, improves the test efficiency and reduces the test cost.

[0119] In one or more embodiments of the present specification, in order to expand the test data, the system parameters of each handling equipment pre-acquired in step S100 may include not only the parameters of each handling equipment and each sensor parameter, but also each posture transformation strategy and / or each simulation scene parameter. Thus, different test cases generated in step S102 may include different posture transformation strategies and / or different simulation scene parameters, and the simulated vehicle point cloud may be obtained when the simulated vehicle model is in different postures relative to the simulated handling equipment model and / or in different simulation scenes.

[0120] The posture transformation strategy is used to indicate that the posture of the simulated handling equipment model or the posture of the simulated vehicle model is transformed in the simulation coordinate system, so as to determine the reference relative posture of the simulated vehicle model relative to the simulated handling equipment model in the simulation coordinate system, thereby obtaining different reference relative postures of the simulated vehicle model relative to the simulated handling equipment model by changing the posture of the simulated handling equipment model or the posture of the simulated vehicle model, obtaining the simulated vehicle point cloud under different reference relative postures, and using it to test the perception algorithm to be tested. Generally, the posture transformation strategy may include a first posture transformation strategy for transforming the posture of the simulated handling equipment model, and a second posture transformation strategy for transforming the posture of the simulated vehicle model. Of course, the posture transformation strategy may also include a transformation strategy for transforming both the posture of the simulated handling equipment model and the posture of the simulated vehicle model. Compared with the current testing process in which the posture of the physical carrier relative to the physical handling equipment is manually measured as the actual posture, this manual automatically adjusts the simulated vehicle model and the simulated handling equipment model through a posture transformation strategy to obtain a reference relative posture. While improving the efficiency of determining the reference relative posture, it ensures the accuracy of the reference relative posture, avoids posture measurement errors caused by manual measurement, and improves the stability and efficiency of the simulation test.

[0121] The simulation scene parameters are used to describe the simulated working environment where the simulated vehicle model is located. The simulation scene parameters may include specific test requirements in the simulated working scene, such as the fork lifting height, and the simulated working scene attribute information, such as the attribute information and terrain characteristics of the simulated ground where the simulated vehicle model and the simulated handling equipment model are located, the climate and lighting conditions in the simulated working environment, and the object parameters of static obstacles or dynamic obstacles contained in the simulated working environment.

[0122] Based on this, step S102 can actually be: determine each test case according to each simulation vehicle model file, and each transport equipment parameter, each sensor parameter, each simulation scene parameter, and each posture transformation strategy contained in each transport equipment system parameter. Considering that the generation process of the simulated vehicle point cloud needs to simulate the real process of the real point cloud, in addition to the simulation vehicle model file, at least the transport equipment parameters and sensor parameters in the transport equipment system parameters are required. Therefore, in an optional embodiment of the present specification, step S102 can actually be: determine each test case at least according to each simulation vehicle model file, and each transport equipment parameter and each sensor parameter contained in each transport equipment system parameter. The simulation scene parameters and each posture transformation strategy contained in each transport equipment system parameter are further optional options based on the above. Among them, different test cases in each test case contain at least one different simulation vehicle model file, transport equipment parameter, sensor parameter, simulation scene parameter, and posture transformation strategy. Figure 1 The step S102 of determining each test case can be implemented by referring to the following implementation method:

[0123] The first step: selecting at least one simulation vehicle model file from the simulation vehicle model files, and selecting at least one set of transport equipment system parameters from the transport equipment system parameters.

[0124] Specifically, the back-end management system of the test platform can automatically select at least one simulation vehicle model file from each simulation vehicle model file according to the preset selection rules based on each simulation vehicle model file and each handling equipment system parameter acquired in advance. Afterwards, at least one group of handling equipment system parameters are automatically selected from each handling equipment system parameter acquired in advance according to the same preset selection rules, and the group of handling equipment system parameters at least includes at least one group of handling equipment parameters selected from each handling equipment parameter, and at least one group of sensor parameters selected from each sensor parameter. On this basis, optionally, at least one group of simulation scene parameters can be automatically selected from each simulation scene parameter according to the same preset selection rules, and / or at least one group of posture transformation strategies can be automatically selected from each posture transformation strategy. Then, based on the selected simulation vehicle model file and the selected handling equipment system parameter, the test task or test case is determined.

[0125] Among them, the pre-set selection rules used for selecting the above files or parameters can be pre-set by the user according to specific test requirements and test scenarios, such as the selection rules are to select in order from early to late according to the entry time, or to select in order from few to many times selected, or all files and parameters obtained by the test platform can be used as selected files and selected parameters, and this manual does not limit this.

[0126] In addition, the test platform can also display various simulation vehicle model files and various handling equipment system parameters pre-acquired by the test platform to the user through the front-end interactive system, and guide the user to make a selection, which can be specifically achieved through the following implementation methods:

[0127] First, the front-end interactive system of the test platform displays the task building interface that includes task generation controls.

[0128] Optionally, in addition to the task generation controls, the task building interface can also display at least part of the information of the generated test task, such as the task identifier, the simulation vehicle model file contained in the test task, the handling equipment parameters, the sensor parameters, the simulation scene parameters and the posture transformation strategy, thereby prompting the user to select different files and parameters, so as to avoid the test task generated based on the user's selection being exactly the same as the generated test task.

[0129] Secondly, the front-end interactive system responds to the user's operation on the task generation control and jumps to display the vehicle selection interface, such as Figure 3 The figure shows an optional schematic diagram of a vehicle selection interface, which displays the identifiers of various simulation vehicle model files pre-acquired by the test platform, and includes a vehicle selection control marked with the guide text "Select" and a first jump control marked with the guide text "Next". In response to the user's operation on the vehicle selection control, at least one simulation vehicle model file selected from the displayed simulation vehicle model files is determined. In addition, the vehicle selection interface also includes a vehicle adding control, which displays the information of each simulation vehicle model file that has been entered, and a file import control, through which the user can input other simulation vehicle model files that are different from the simulation vehicle model files that have been entered.

[0130] Afterwards, in response to the user's operation on the first jump control, the display jumps to the handling equipment and sensor parameter selection interface, such as Figure 4 The figure shows an optional schematic diagram of a handling equipment and sensor parameter selection interface, wherein the handling equipment and sensor parameter selection interface displays various handling equipment parameters and various sensor parameters pre-acquired by the test platform, and includes a handling equipment parameter selection control marked with a guide text "Select", a sensor parameter selection control marked with a guide text "Select", and a second jump control marked with a guide text "Next". In response to the user's operation on the handling equipment parameter selection control, at least one group of handling equipment parameters selected from the displayed handling equipment parameters is determined. In response to the user's operation on the sensor parameter selection control, at least one group of sensor parameters selected from the displayed sensor parameters is determined. In addition, the handling equipment and sensor parameter selection interface may also include a handling equipment parameter adding control and a sensor parameter adding control, through which the user can input other handling equipment parameters different from the various handling equipment parameters that have been entered, and through which the user can input other sensor parameters different from the various sensor parameters that have been entered.

[0131] Then, in response to the user's operation on the second jump control, the jump display strategy and scene selection interface is displayed, such as Figure 5The diagram shows an optional schematic diagram of a strategy and scenario selection interface, wherein the strategy and scenario selection interface displays various posture transformation strategies and various simulation scenario parameters pre-acquired by the test platform, and includes a posture transformation strategy selection control marked with the guiding text "Select" and a simulation scenario parameter selection control marked with the guiding text "Select". In response to the user's operation on the posture transformation strategy selection control, at least one group of posture transformation strategies selected from the displayed posture transformation strategies is determined. In response to the user's operation on the simulation scenario parameter selection control, at least one group of simulation scenario parameters selected from the displayed simulation scenario parameters is determined. In addition, the strategy and scenario selection interface may also include a simulation scenario parameter adding control, through which the user can input other simulation scenario parameters that are different from the various simulation scenario parameters that have been entered.

[0132] Step 2: Generate test tasks based on the selected simulation vehicle model file and the selected handling equipment system parameters.

[0133] The front-end interactive system of the test platform can send information of the selected simulation vehicle model file and information of the selected handling equipment system parameters to the back-end management system, and the back-end management system generates a test task based on the selected simulation vehicle model file and the selected handling equipment system parameters.

[0134] Optionally, a third jump control marked with a guiding text "next step" is displayed on the strategy and scenario selection interface through the front-end interactive system. In response to the user's operation on the third jump control, the task generation interface is jumped and displayed, such as Figure 6 FIG. 1 is an optional schematic diagram of a task generation interface, in which the selected simulation vehicle model file a is displayed. 1 、Selected handling equipment parameters b 1 , selected sensor parameters c 2 and c 3 , selected simulation scenario parameters e 2 And the selected pose transformation strategy d 1 In response to the confirmation operation input by the user, a test task generation request is determined according to the selected simulation vehicle model file, the selected handling equipment parameters, the selected sensor parameters, the selected simulation scene parameters, and the selected posture transformation strategy, and the test task generation request is sent to the back-end management system. The back-end management system responds to the test task generation request and generates a test task according to the simulation vehicle model file, handling equipment parameters, sensor parameters, simulation scene parameters, and posture transformation strategy parsed from the test task generation request.

[0135] Step 3: Arrange and combine the simulation vehicle model files and handling equipment system parameters included in the test task to obtain test cases corresponding to the test task.

[0136] Among them, different test cases corresponding to the same test task respectively contain at least one difference in the simulation vehicle model file, handling equipment parameters, sensor parameters, simulation scene parameters and posture transformation strategy.

[0137] Specifically, assuming that a test task contains multiple simulation vehicle model files, multiple sets of handling equipment parameters, multiple sets of sensor parameters, multiple sets of simulation scene parameters, and multiple posture transformation strategies, the execution of the test task description requires testing the perception algorithm to be tested in multiple simulation scenes and with different posture transformation strategies based on multiple different simulation vehicle models, multiple different simulation handling equipment models, and multiple simulation sensors. Therefore, various test situations corresponding to the test task can be traversed by permutation and combination, thereby generating test cases corresponding to the test task.

[0138] Therefore, in this specification, the determined test case includes a simulation vehicle model file and a handling equipment system parameter, and the handling equipment system parameter in the test case may at least include handling equipment parameters and sensor parameters, so as to construct a simulation vehicle model and a simulation handling equipment model in a simulation coordinate system, and deploy a simulation sensor on the simulation handling equipment model. On this basis, the handling equipment system parameters in the test case may optionally include simulation scene parameters and / or posture transformation strategies. Thus, there are the following four situations for the handling equipment system parameters included in the test case: the first is that the handling equipment system parameters in the test case include handling equipment parameters and sensor parameters; the second is that the handling equipment system parameters in the test case include handling equipment parameters, sensor parameters and simulation scene parameters; the third is that the handling equipment system parameters in the test case include handling equipment parameters, sensor parameters and posture transformation strategies. The fourth is the test case handling equipment parameters, sensor parameters, simulation scene parameters and posture transformation strategies.

[0139] In addition, different test cases contain at least one difference in the simulation vehicle model file, handling equipment parameters, sensor parameters, simulation scene parameters and posture transformation strategy, so as to avoid generating the same simulation vehicle point cloud based on the same test case, which would cause a waste of computing resources.

[0140] For example, test case X 1 Contains simulation vehicle model file a 1 , handling equipment parameters b 1 、Sensor parameters c 2 and c 3 , simulation scene parameters e 2 And the pose transformation strategy d 1 , test case X 2 Contains simulation vehicle model file a 3, handling equipment parameters b 2 、Sensor parameters c 1 , simulation scene parameters e 2 And the pose transformation strategy d 1 , then test case X 1 and test case X 2 The simulation vehicle model files, handling equipment parameters and sensor parameters are different, and only the posture transformation strategy and simulation scene parameters are the same. However, if there is a test case X 3 , which contains the simulation vehicle model file a 1 , handling equipment parameters b 1 、Sensor parameters c 2 and c 3 , simulation scene parameters e 2 And the pose transformation strategy d 1 , obviously test case X 1 and test case X 3 The files and parameters contained are exactly the same. Test case X can be deleted at this time. 1 Or delete test case X 3 to avoid duplication of tests.

[0141] In an optional embodiment of the present specification, since the above-mentioned scheme of determining the test tasks and generating each test case by permutation and combination may result in too many test cases, testing the perception algorithms to be tested one by one based on too many test cases can improve the coverage and comprehensiveness of the test, but not only the test execution time is too long and the test resources are wasted, but there may also be problems such as increased test case maintenance costs and difficulty in analyzing a large number of simulation test results. Therefore, the above-mentioned step S102 can be implemented according to the following implementation method:

[0142] First, in response to the user's selection operation, a selected simulation vehicle model file is determined from the simulation vehicle model files, and at least one set of selected transport equipment system parameters is determined from the transport equipment system parameters; wherein a set of selected transport equipment system parameters includes at least one set of transport equipment parameters and at least one set of sensor parameters.

[0143] Specifically, the front-end interactive system of the test platform can guide the user to select specific simulation vehicle model files and handling equipment system parameters, such as one or more of the handling equipment parameters, sensor parameters, simulation scene parameters, and posture transformation strategies in the handling equipment system parameters, by displaying the test case customization interface, so as to generate a small number of test cases that meet the test requirements. In this way, the number of test cases is reduced and the test requirements are covered, thereby improving the test efficiency and reducing the test cost. For example, the front-end interactive system of the test platform can display the test case customization interface, and a custom case control is displayed on the test case customization interface. The user can interact with the custom case control, such as clicking, to execute the test case customization process.

[0144] Furthermore, the front-end interactive system of the test platform can display the pre-acquired simulation vehicle model files and the system parameters of each handling equipment in the test case customization interface in response to the user's operation on the custom use case control. The displayed system parameters of each handling equipment include at least the parameters of each handling equipment and the parameters of each sensor. In addition, optionally, the displayed system parameters of each handling equipment may also include the parameters of each simulation scene and / or each posture transformation strategy. Each simulation vehicle model file and each handling equipment system parameter can be displayed on one page or on multiple pages respectively, and this specification does not limit this.

[0145] like Figure 7 The figure shows an optional schematic diagram of the test case customization interface. The user can select the simulation vehicle model file and the handling equipment system parameters by selecting the mark in the option box. In response to the user's operation, the front-end interactive system of the test platform can determine the simulation vehicle model file and the handling equipment system parameters selected by the user. Among them, a set of selected handling equipment system parameters includes at least one set of selected handling equipment parameters and at least one set of selected sensor parameters, thereby generating a test case.

[0146] Then, each test case is generated according to the selected simulation vehicle model file and the selected handling equipment system parameters.

[0147] The front-end interactive system can send the selected simulation vehicle model file and the selected handling equipment system parameters to the back-end management system, which generates test cases based on them and records the generated test cases in a test case table pre-established in the database of the back-end management system.

[0148] In one or more embodiments of the present specification, based on the selected simulation vehicle model file and the selected handling equipment system parameters, whether generating a test task or directly generating a test case, the selected handling equipment system parameters determined may include at least one set of simulation scene parameters and / or at least one set of posture transformation strategies in addition to at least one set of handling equipment parameters and at least one set of sensor parameters. By flexibly selecting simulation vehicle model files and handling equipment system parameters, test cases that meet different test requirements can be obtained, thereby performing multi-directional and multi-angle simulation tests on the perception algorithm to be tested, so as to obtain comprehensive and reliable simulation test results of the perception algorithm to be tested.

[0149] In an optional embodiment of the present specification, a test task table and a test case table may be pre-established in the database of the back-end management system of the test platform. Among them, the test task table records the task information corresponding to each test task, and the task information includes the task identification, the identification of the simulation vehicle model file contained in the test task, the handling equipment parameters, the sensor parameters, the simulation scene parameters and the posture transformation strategy. The test case table records the use case information corresponding to each test case, and the use case information includes the use case identification, the task identification of the test task to which the test case belongs, the identification of the simulation vehicle model file contained in the test case, the handling equipment parameters, the sensor parameters, the simulation scene parameters and the posture transformation strategy.

[0150] In one or more embodiments of the present specification, based on the test case including the simulation vehicle model file and the handling equipment system parameters, and the handling equipment system parameters in the test case at least including the handling equipment parameters and the sensor parameters, Figure 1 The step S104 shown can be specifically implemented by the following implementations:

[0151] Step 1: For each test case, determine the initial pose of the simulated vehicle model according to the simulated vehicle model and various vehicle parameters contained in the simulated vehicle model file in the test case.

[0152] Based on the simulation vehicle model and various vehicle parameters contained in the simulation vehicle model file in the test case, a simulation vehicle model can be constructed in the simulation coordinate system. At this time, the pose of the simulation vehicle model can be the default initial pose. The specific position of the initial pose of the simulation vehicle model in the simulation coordinate system can be determined by a preset method, and this specification does not limit the specific parameters of the initial pose. The initial pose of the simulation handling equipment model in the simulation coordinate system is similar, and will not be elaborated here.

[0153] Step 2: Determine the initial pose of the simulated handling equipment model based on the handling equipment parameters in the test case.

[0154] Step 3: Determine a reference relative pose of the simulated handling equipment model relative to the simulated handling equipment model according to the initial pose of the simulated carrier model.

[0155] Furthermore, based on the difference between the initial pose of the simulated vehicle model in the simulation coordinate system and the initial pose of the simulated handling equipment model in the simulation coordinate system, a reference relative pose of the simulated vehicle model relative to the simulated handling equipment model in the simulation coordinate system is determined.

[0156] Optionally, when the handling equipment system parameters in the test case also include a posture transformation strategy, the reference relative posture of the simulated vehicle model relative to the simulated handling equipment model can be determined based on the initial posture of the simulated handling equipment model and the initial posture of the simulated vehicle model under the guidance of the posture transformation strategy. Among them, the posture transformation strategy is used to indicate whether to transform the posture of the simulated handling equipment model or the posture of the simulated vehicle model. Therefore, when the handling equipment system parameters in the test case also include a posture transformation strategy, according to the posture transformation strategy in the test case, the reference relative posture of the simulated vehicle model relative to the simulated handling equipment model can be determined in the following situations:

[0157] The first case: when the posture transformation strategy in the test case is the first posture transformation strategy, the initial posture of the simulated handling equipment model is transformed, and the reference relative posture of the simulated handling equipment model relative to the simulated handling equipment model is determined based on the transformed posture of the simulated handling equipment model and the initial posture of the simulated vehicle model, especially the difference between the transformed posture of the simulated handling equipment model and the initial posture of the simulated vehicle model.

[0158] The second case: when the posture transformation strategy in the test case is the second posture transformation strategy, the initial posture of the simulated vehicle model is transformed, and the reference relative posture of the simulated vehicle model relative to the simulated handling equipment model is determined based on the initial posture of the simulated handling equipment model and the transformed posture of the simulated vehicle model, especially the difference between the initial posture of the simulated handling equipment model and the transformed posture of the simulated vehicle model.

[0159] In addition, the posture transformation strategy can also instruct to transform both the posture of the simulated handling equipment model and the posture of the simulated vehicle model. At this time, the initial posture of the simulated handling equipment model is transformed, and the initial posture of the simulated vehicle model is transformed. According to the transformed posture of the simulated handling equipment model and the transformed posture of the simulated vehicle model, especially the difference between the transformed posture of the simulated handling equipment model and the transformed posture of the simulated vehicle model, the reference relative posture of the simulated vehicle model relative to the simulated handling equipment model is determined.

[0160] Step 4: Generate a simulated vehicle point cloud corresponding to the test case based on the sensor parameters in the test case and the reference relative pose.

[0161] This step is similar to the aforementioned step S104 and will not be described again here.

[0162] Optionally, if the handling equipment system parameters in the test case also include simulation scene parameters, a simulation scene can be constructed according to the simulation scene parameters in the test case, and a simulated vehicle point cloud corresponding to the test case is generated in the constructed simulation scene. That is, when the test case includes simulation scene parameters, a simulated vehicle point cloud corresponding to the test case is generated according to the sensor parameters and simulation scene parameters in the test case, as well as the reference relative pose of the simulated vehicle model relative to the simulated handling equipment model.

[0163] In this manual, Figure 1 Step S106 can be specifically implemented by the following implementations:

[0164] The first step: input the simulated vehicle point cloud corresponding to the target test case into the perception algorithm to be tested, and through the perception algorithm to be tested, identify the target relative posture of the simulated vehicle model in the target test case relative to the simulated handling equipment model from the simulated vehicle point cloud corresponding to the target test case.

[0165] Specifically, the perception algorithm to be tested can extract key features that can represent the shape and structural information of the object from the simulated vehicle point cloud, identify the simulated vehicle model and the simulated handling equipment model based on the extracted features, and obtain the estimated pose of the simulated vehicle model and the estimated pose of the simulated handling equipment model in the simulation coordinate system, thereby obtaining the target relative pose of the simulated vehicle model relative to the simulated handling equipment model. The above is only an optional embodiment of the perception algorithm to be tested to obtain the target relative pose, and the implementation method of the perception algorithm to be tested in this specification to identify the vehicle pose is not limited to this.

[0166] Step 2: Using the reference relative posture of the simulated vehicle model relative to the simulated handling equipment model in the target test case as a reference, and determining the simulation test result of the perception algorithm to be tested according to the difference between the target relative posture and the reference relative posture.

[0167] Afterwards, the reference relative pose of the simulated vehicle model relative to the simulated handling equipment model in the target test case is taken as a reference, that is, the true value, and the target relative pose identified by the perception algorithm to be tested is taken as an estimated value to determine the difference between the target relative pose and the reference relative pose, and the simulation test result of the perception algorithm to be tested is determined based on the difference.

[0168] In one or more embodiments of the present specification, a test case may correspond to multiple groups of simulated vehicle point clouds, and different groups of simulated vehicle point clouds respectively correspond to different reference relative poses of the simulated vehicle model relative to the simulated handling equipment model. Figure 1 In step S104, for the same test case, with the same simulated vehicle model, the same simulated handling equipment model and the same simulated sensor, only the reference relative pose of the simulated vehicle model relative to the simulated handling equipment model is changed, and multiple groups of simulated vehicle point clouds are obtained under multiple different reference relative poses.

[0169] Therefore, in one or more embodiments of this specification, Figure 1 Step S106 can be specifically implemented by the following implementations:

[0170] The first step: for each group of simulated vehicle point clouds corresponding to the target test case, the group of simulated vehicle point clouds is input into the perception algorithm to be tested, and the target relative posture of the simulated vehicle model corresponding to the group of simulated vehicle point clouds relative to the simulated handling equipment model is identified from the group of simulated vehicle point clouds through the perception algorithm to be tested.

[0171] The perception algorithm to be tested is used to identify each group of simulated vehicle point clouds corresponding to the target test case one by one. The identification process can be referred to as above and will not be repeated here.

[0172] Step 2: Taking the reference relative pose of the simulated vehicle model corresponding to the group of simulated vehicle point clouds relative to the simulated handling equipment model as a reference, and determining the simulation test result of the perception algorithm to be tested detecting the group of simulated vehicle point clouds based on the difference between the reference relative pose corresponding to the group of simulated vehicle point clouds and the target relative pose corresponding to the group of simulated vehicle point clouds.

[0173] Step 3: Detect the simulation test results of each group of simulated vehicle point clouds corresponding to the target test case according to the perception algorithm to be tested, and determine the simulation test results of the perception algorithm to be tested.

[0174] By integrating the simulation test results corresponding to each group of simulated vehicle point clouds, the recognition accuracy and precision of the perception algorithm to be tested under different reference relative postures are analyzed to provide direction for the adjustment and optimization of the perception algorithm to be tested.

[0175] Optionally, different groups of simulated vehicle point clouds correspond to different test distances, where the test distance is the distance between the vehicle fork entry surface of the simulated vehicle model and the vehicle body center of the simulated handling equipment model. Figure 1 In the shown step S104, for the same test case, with the same simulated vehicle model, the same simulated handling equipment model and the same simulated sensor, the reference relative posture of the simulated vehicle model relative to the simulated handling equipment model is changed so that the distance between the vehicle entry surface of the simulated vehicle model and the vehicle body center of the simulated handling equipment model is changed, thereby obtaining multiple groups of simulated vehicle point clouds under the conditions of multiple different test distances.

[0176] Therefore, during the test process, the target relative posture of the simulated vehicle model relative to the simulated handling equipment model is identified from each group of simulated vehicle point clouds according to the perception algorithm to be tested. The simulation test results of the perception algorithm to be tested under this group of simulated vehicle point clouds can characterize the accuracy and precision of the perception algorithm to be tested in identifying this group of simulated vehicle point clouds at the test distance corresponding to this group of simulated vehicle point clouds.

[0177] Furthermore, based on the simulation test results of the perception algorithm to be tested respectively detecting each group of simulated vehicle point clouds corresponding to the target test case, the simulation test results of the perception algorithm to be tested determined actually represent the precision distribution and accuracy distribution of the perception algorithm to be tested in identifying the pose of the simulated vehicle model at multiple different test distances corresponding to each group of simulated vehicle point clouds included in the target test case. Since the test distance will affect the result of the perception algorithm identifying the pose of the vehicle, based on the simulation test results of the perception algorithm to be tested identifying the pose of the vehicle from each group of simulated vehicle point clouds at different test distances, the precision and accuracy of the perception algorithm to be tested in identifying the pose of the vehicle at different test distances can be verified.

[0178] In an optional embodiment of the present specification, the perception algorithm to be tested includes multiple perception algorithms, and different perception algorithms detect different features of the vehicle when identifying the vehicle posture.

[0179] thereby, Figure 1 The step S106 shown can be specifically implemented by the following implementations:

[0180] First, for each perception algorithm to be tested, the simulated vehicle point cloud corresponding to the target test case is input into the perception algorithm to be tested, and the perception algorithm to be tested detects the characteristics of the vehicle and, from the simulated vehicle point cloud corresponding to the target test case, identifies the relative pose of the simulated vehicle model in the target test case relative to the simulated handling equipment model as the target relative pose output by the perception algorithm to be tested.

[0181] Among them, the perception algorithms to be tested may include a pallet detection algorithm, a tight pallet detection algorithm, a pier hole detection algorithm, a socket detection algorithm and a cage detection algorithm. The pallet detection algorithm can determine the pallet posture by detecting the edge, contour, height and other features of the pallet. The tight pallet detection algorithm can determine the pallet posture by detecting the spacing between pallets, overlapping areas, edge details and other features. The pier hole detection algorithm can identify the carrier posture by detecting the carrier foot piers, the socket detection algorithm can identify the carrier posture by detecting the carrier sockets, and the cage detection algorithm can identify the cage posture by detecting the cage frame structure, grid and other features. In short, different perception algorithms to be tested may differ in identifying the characteristics of the carrier, but the results of the algorithm output are all the posture of the carrier relative to the handling equipment.

[0182] Then, the reference relative posture of the simulated vehicle model relative to the simulated handling equipment model in the target test case is used as a reference, and the simulation test result of the perception algorithm to be tested is determined according to the difference between the target relative posture output by the perception algorithm to be tested and the reference relative posture.

[0183] Based on the above scheme, the simulation test results of different perception algorithms under the same simulation test conditions can be obtained, so as to analyze the detection accuracy and precision of each perception algorithm to be tested by detecting different vehicle characteristics, and provide data support for the adjustment and optimization of the perception algorithm to be tested and the selection of perception algorithms in practical applications.

[0184] In an optional embodiment of this specification, in Figure 1 On the basis of the simulation test shown, the precision and accuracy of the perception algorithm to be tested can also be comprehensively evaluated in combination with the real test results obtained from the real test, thereby improving the accuracy of the test. The real test scheme for the perception algorithm to be tested can refer to the following specific implementation methods:

[0185] Step 1: Pre-acquire the real vehicle point cloud corresponding to the target test case.

[0186] Since the simulated vehicle point cloud is generated by simulation in the test platform simulation coordinate system, rather than collected by physical sensors, the simulated vehicle point cloud is usually evenly distributed and has the best performance. However, in actual applications, the point cloud data input into the perception algorithm to be tested is obtained by scanning the physical sensor in the real working environment. Since the physical sensor will be affected by various external factors during production, transportation, installation, and use, the point cloud data generated by scanning the environment with the physical sensor may have noise, missing points, tailing, and other problems, so there will be differences between the real vehicle point cloud and the simulated vehicle point cloud.

[0187] Therefore, the simulation test of the perception algorithm to be tested based on the simulated vehicle point cloud is actually testing the precision and accuracy of the perception algorithm to be tested in an ideal environment. However, the simulated vehicle point cloud is too idealized and cannot fully reflect the complexity and uncertainty in the actual working environment. It also ignores the details of the physical sensor. Therefore, the precision and accuracy of the perception algorithm to be tested can be comprehensively evaluated by combining the simulation test with the real test.

[0188] First, the real vehicle point cloud corresponding to the target test case is obtained in advance. The specific implementation method of the acquisition can be as follows: construct an actual test scene that meets the simulation scene parameters in the target test case. According to the simulation vehicle model file, handling equipment parameters and sensor parameters contained in the target test case, in the constructed actual test scene, place a physical vehicle that is the same or similar to the simulation vehicle model in the simulation vehicle model file, place a physical handling equipment that meets the handling equipment parameters, and place a physical sensor that meets the sensor parameters on the physical handling equipment. Adjust the pose of the physical vehicle and the pose of the physical handling equipment with the reference relative pose of the simulation vehicle model corresponding to the simulation vehicle point cloud corresponding to the target test case relative to the simulation handling equipment model, so that the actual relative pose of the physical vehicle relative to the physical handling equipment is the same as the reference relative pose of the simulation vehicle model corresponding to the simulation vehicle point cloud relative to the simulation handling equipment model. Afterwards, under the above conditions, the real vehicle point cloud corresponding to the target test case is obtained by scanning the physical vehicle through the physical sensor.

[0189] Step 2: In response to the real test request, the perception algorithm to be tested is tested according to the real vehicle point cloud corresponding to the target test case to obtain the real test result of the perception algorithm to be tested.

[0190] Similar to the aforementioned step S102, the real vehicle point cloud corresponding to the target test case is input into the perception algorithm to be tested. The perception algorithm to be tested can identify the physical vehicle from the real vehicle point cloud and determine the estimated pose of the physical vehicle relative to the physical handling equipment. Based on the difference between the estimated pose output by the perception algorithm to be tested and the actual relative pose of the physical vehicle relative to the physical handling equipment, the actual test result of the perception algorithm to be tested is determined.

[0191] Since the real vehicle point cloud comes from the physical sensor scanning the physical vehicle in the actual test scenario, it can truly reflect the uncertainty of the physical sensor and the complexity of the actual test scenario. Therefore, the real test results can reflect the recognition accuracy and precision of the perception algorithm to be tested in the real environment, thereby verifying the performance of the perception algorithm to be tested in the real environment.

[0192] Optionally, based on any type of existing statistical method, the simulation test results and the real test results of the perception algorithm to be tested can be statistically analyzed respectively to obtain evaluation indicators for characterizing the precision and accuracy of the perception algorithm to be tested during the simulation test, and to obtain evaluation indicators for characterizing the precision and accuracy of the perception algorithm to be tested during the real test, and the evaluation indicators of the simulation test and the evaluation indicators of the real test can be combined to generate a test report corresponding to the perception algorithm to be tested. The specific statistical method and test report generation scheme can refer to the following implementation method for generating a test report, which will not be described in detail here.

[0193] It should be noted that whether to conduct a real test can be flexibly selected based on factors such as the problems existing in the actual work of the perception algorithm to be tested, the test requirements, or the simulation test results of the perception algorithm to be tested. For example, when the perception algorithm to be tested performs the task of identifying the vehicle posture in the actual working environment, it may be unable to normally identify the vehicle posture due to the complex situation of the actual working environment, such as the presence of unconventional obstacles. At this time, the user can collect the real vehicle point cloud under the actual working environment and record the actual relative posture of the physical vehicle relative to the physical handling equipment under the actual working environment, and conduct a real test on the perception algorithm to be tested based on the real vehicle point cloud. In addition, based on the actual working environment, the physical vehicle, the physical handling equipment, and the physical sensor under the actual working environment, add a simulation vehicle model file with the same model and parameters to the test platform, and add the corresponding handling equipment parameters, sensor parameters, and simulation scene parameters to determine the test case, and generate a simulation vehicle point cloud, so as to perform a simulation test on the perception algorithm to be tested. Then, combined with the real test results and the simulation test results, the perception algorithm to be tested is comprehensively evaluated to determine the reasons for the recognition failure of the perception algorithm to be tested, so as to make targeted adjustments and optimizations to the perception algorithm to be tested.

[0194] Step 3: Determine the comprehensive test result of the perception algorithm to be tested according to the simulation test result and the actual test result.

[0195] Afterwards, the simulation test results and the actual test results of the same perception algorithm to be tested are combined to obtain a comprehensive test result. The comprehensive test result may include a fusion and comparison of the performance of the perception algorithm to be tested in the simulation coordinate system and the performance in the real environment, so as to determine the possible problems of the perception algorithm to be tested in different environments, and use this to improve the perception algorithm to be tested, so as to enhance the performance of the perception algorithm to be tested in the actual environment.

[0196] In addition, in the comprehensive test results, the proportion of the simulation test results and the proportion of the real test results may be the same or different, and this specification does not limit this.

[0197] This specification does not limit the test order between the simulation test based on the simulated vehicle point cloud and the real test based on the real vehicle point cloud.

[0198] In an optional embodiment of this specification, Figure 1 The simulation vehicle model file, each handling equipment parameter, each sensor parameter and each simulation scene parameter pre-acquired in step S100 can be pre-acquired by the test platform and stored in the database of the backend management system, in which a vehicle management table, a handling equipment parameter management table, a sensor parameter management table, a simulation scene parameter management table and a perception algorithm management table are pre-established. Among them, the vehicle management table is used to store each vehicle parameter and simulation vehicle model thumbnail of the entered simulation vehicle model file, the handling equipment parameter management table is used to store each group of handling equipment parameters that have been added, the sensor parameter management table is used to store each group of sensor parameters that have been added, the simulation scene parameter management table is used to store each group of simulation scene parameters that have been added, and the perception algorithm management table is used to record each perception algorithm to be tested.

[0199] The simulation vehicle model file, the parameters of each handling device and the parameters of each sensor can be obtained by the test platform using any one of the following two implementations.

[0200] The first type: The test platform obtains the product description document of the existing physical product in advance and determines the model or parameters based on it.

[0201] Specifically, the product description documents corresponding to each physical carrier, each physical handling equipment, and each physical sensor can be obtained in advance. The product model of the physical carrier is extracted from the product description document of the physical carrier and used to construct the simulation vehicle model corresponding to the physical carrier, and the product parameters of the physical carrier are extracted and used to determine the various vehicle parameters of the simulation vehicle model, and then a simulation vehicle model file is generated based on the simulation vehicle model and the various vehicle parameters. Similarly, the handling equipment parameters of the physical handling equipment can be extracted from the product description document of the physical handling equipment and stored, and the sensor parameters of the physical sensor can be extracted from the product description document of the physical sensor. In this way, the back-end management system of the test platform can obtain the models or parameters of any type of existing carriers, handling equipment, and sensors, thereby improving the coverage of the simulation test.

[0202] The second method: users input models or parameters through the front-end interactive system of the test platform.

[0203] Since simulation testing is flexible, the models and parameters used in simulation testing may not be restricted by actual conditions. Therefore, the models or parameters pre-acquired by the test platform can be customized by the user. Take the example of a user inputting a simulation vehicle model file through the front-end interactive system of the test platform. Specifically, the user can determine the simulation vehicle model according to the specific application scenario and simulation test requirements, such as the three-dimensional model of the simulation vehicle model, and determine the size of the three-dimensional structure of the simulation vehicle model. Afterwards, the user can input the simulation vehicle model and the size of the simulation vehicle model into the front-end interactive system through the terminal device in the front-end interactive system of the test platform, so that the front-end interactive system can send the simulation vehicle model and the size of the simulation vehicle model input by the user to the back-end management system, and the back-end management system can generate the simulation vehicle model file and store and manage it. The same is true for handling equipment parameters and sensor parameters.

[0204] It is understandable that, since the simulation scene parameters usually do not correspond to the physical product, the simulation scene parameters stored in the database are usually pre-acquired by the test platform based on the second implementation method described above.

[0205] Optionally, the specific implementation method of the user inputting the simulation vehicle model file, handling equipment parameters, sensor parameters and simulation scene parameters into the front-end interactive system through the terminal device in the front-end interactive system of the test platform can be as follows:

[0206] The front-end interactive system of the test platform displays the simulation task main interface, which includes option management controls, including vehicle management controls that jump to the simulation vehicle model file management interface, transport equipment parameter management controls that jump to the transport equipment parameter management interface, and sensor parameter management controls that jump to the sensor parameter management interface.

[0207] In response to the user's operation on the vehicle management control, the simulation vehicle model file management interface is displayed, such as Figure 8 The figure shows an optional schematic diagram of a simulation vehicle model file management interface, in which a file import control is displayed, as well as thumbnails of simulation vehicle models corresponding to each input simulation vehicle model file and corresponding vehicle parameters. In response to the user's operation on the file import control, the simulation vehicle model file input by the user is obtained.

[0208] In response to the user's operation on the handling equipment parameter management control, the handling equipment parameter management interface is displayed, such as Fig. 9 The figure shows an optional schematic diagram of a handling equipment parameter management interface, in which a handling equipment parameter adding control and each group of handling equipment parameters that have been entered are displayed. Each group of handling equipment parameters may include: Fig. 9 The dimensions of the handling equipment, the dimensions of the attachments, the height of the legs and the mast posture of the handling equipment shown in . In response to the operation of adding a control to the parameters of the handling equipment by the user, the parameters of the handling equipment input by the user are obtained.

[0209] In response to the user's operation on the sensor parameter management control, the sensor parameter management interface is displayed, such as Fig.10 The figure shows an optional schematic diagram of a sensor parameter management interface, in which the sensor parameter management interface displays a sensor parameter adding control and each group of sensor parameters that have been entered. Each group of sensor parameters may include the following: Fig.10 The sensor type, internal parameters and external parameters shown. In response to the user's operation of adding a control to the sensor parameter, the sensor parameter input by the user is obtained.

[0210] In response to the user's operation on the simulation scene parameter management control, the simulation scene parameter management interface is displayed. Fig.11 The figure shows an optional schematic diagram of a simulation scene parameter management interface, in which a simulation scene parameter adding control and each group of simulation scene parameters that have been entered are displayed. Each group of simulation scene parameters may include: Fig.11 The fork lifting height and working scene attributes shown. In response to the user's operation of adding controls to the simulation scene parameters, the simulation scene parameters input by the user are obtained.

[0211] In addition, the user can input the simulation vehicle model and its size in the third-party software. The simulation vehicle model can be a model of any shape, and the size of the simulation vehicle model can be any size, which is not limited in this specification. The third-party software can generate a simulation vehicle model file based on the simulation vehicle model and its size input by the user. After that, the user can import the simulation vehicle model file generated in the third-party software into the test platform, so that the test platform obtains the simulation vehicle model file. Among them, the third-party software can be any type of existing model building software, such as SolidWorks, CAD, ECAD, etc. This specification does not specifically limit the type of third-party software and the file format of the simulation vehicle model file imported into the test platform.

[0212] Therefore, based on the above-mentioned scheme that users can input the simulation vehicle model and various vehicle parameters into the third-party software, users can customize the simulation vehicle model of any shape and any size for simulation testing. In this way, for the case where the physical vehicle is not in the actual test site, the user can define a simulation vehicle model that is the same in shape and size as the physical vehicle through the third-party software based on the actual shape and actual size of the physical vehicle. As a result, there is no need to conduct testing based on the physical vehicle, and it is not restricted by whether the physical vehicle is in the actual test site, thereby improving the test efficiency.

[0213] In an optional embodiment of this specification, in Figure 1 After obtaining the simulation test results of the perception algorithm to be tested in the illustrated implementation mode, the simulation test results can be further analyzed to obtain a test report corresponding to the perception algorithm to be tested, so as to intuitively and simply describe and illustrate the performance of the perception algorithm to be tested in the simulation test. The specific implementation mode is as follows:

[0214] The first step is to collect statistics on the simulation test results of the perception algorithm to be tested, and determine the accuracy of the perception algorithm to be tested in detecting the pose of the simulated vehicle model corresponding to the target test case.

[0215] Specifically, the accuracy of the posture of the simulated vehicle model corresponding to the target test case detected by the perception algorithm to be tested can characterize the stability of the posture recognition of the perception algorithm to be tested. High accuracy means that the differences between the results of multiple detections of the perception algorithm to be tested under the same conditions are small.

[0216] In addition, the accuracy of the posture of the simulated vehicle model corresponding to the target test case detected by the perception algorithm to be tested can also be determined based on the simulation test results. The accuracy can characterize the similarity between the posture identified by the perception algorithm to be tested and the reference relative posture. High accuracy means that the error between the posture identified by the perception algorithm to be tested and the reference relative posture is small.

[0217] Based on any type of existing statistical methods, evaluation indicators for characterizing the detection precision of the perception algorithm to be tested and evaluation indicators for detection accuracy can be obtained, so as to objectively evaluate the precision and accuracy of the perception algorithm to be tested in identifying the pose of the simulated vehicle model, and provide indicator basis for the optimization and improvement of the perception algorithm to be tested. The statistical method can be a descriptive statistical method, such as mean, standard deviation, maximum value, etc., or a frequency distribution method, such as histogram, cumulative distribution function, etc., or a hypothesis testing method, such as T test. The statistical method can be flexibly selected according to the specific application scenario and test analysis requirements. This manual does not limit the type and number of statistical methods used.

[0218] Step 2: Detect the accuracy of the posture of the simulation vehicle model corresponding to the target test case according to the determined perception algorithm to be tested, and generate a test report corresponding to the perception algorithm to be tested.

[0219] In addition, the precision and accuracy of the perception algorithm to be tested obtained based on statistical analysis can be visualized, such as drawing scatter plots, histograms, box plots and other visual charts, and generating a test report corresponding to the perception algorithm to be tested, so as to intuitively display the distribution and characteristics of the precision and accuracy of the perception algorithm to be tested, thereby assisting users to comprehensively evaluate the perception algorithm to be tested and improve testing efficiency.

[0220] In an optional embodiment of this specification, in Figure 1 In the process of using the target test case to simulate the perception algorithm to be tested in step S106, the backend management system of the test platform can periodically monitor whether the simulation test based on the target test case is completed. If completed, the front-end interactive system of the test platform can inform the user. In addition, since the time spent on testing the perception algorithm to be tested based on different test cases is usually similar, the backend management system can also determine whether the test has timed out based on prior test experience, so as to promptly discover abnormal simulation tests. The specific implementation method can be as follows:

[0221] First, obtain the preset query cycle and the preset test timeout.

[0222] As mentioned above, you can query whether the simulation test results exist at regular intervals. If they exist, it means that the simulation test of the perception algorithm to be tested using the target test case has been completed normally. This information can be fed back to the user to promptly remind the user to view or analyze the simulation test results. If the query does not exist, it means that the simulation test has not yet ended and further waiting is required. If the simulation test results cannot be queried for a long time, it is likely that the simulation test is abnormal. In this case, the information that the simulation test may be abnormal can be fed back to the user in a timely manner so that the user can handle the abnormal test in a timely manner.

[0223] Based on this, the preset query cycle is used to characterize the time interval between two adjacent queries. For example, if the preset query cycle is 10s, it means that every 10s, it can be queried whether there is a simulation test result of the perception algorithm to be tested obtained by executing the target test case. The preset test timeout is used to indicate the maximum allowed time to wait for the simulation test to complete. If the simulation test result is not found after the preset test timeout, it means that the simulation test is not completed within a certain period of time, and there may be abnormalities and other problems. For example, if the preset timeout is 60s, then after 60s from the start of the simulation execution, if the simulation test result of the perception algorithm to be tested obtained by executing the target test case is not found, it can be considered that the simulation test is abnormal.

[0224] Secondly, whenever the query cycle is reached, query whether there is a simulation test result of the perception algorithm to be tested obtained by executing the target test case. If yes, update the test status of the target test case to completed, and show the user the completed test status of the target test case. If not, continue to wait.

[0225] Then, when the preset test timeout is reached, if the simulation test result of the perception algorithm to be tested obtained by executing the target test case does not exist, the test status of the target test case is updated to abnormal, and the abnormal test status of the target test case is displayed to the user.

[0226] Specifically, when the query cycle is reached, if the query result is that there is a simulation test result of the perception algorithm to be tested obtained by executing the target test case, it means that the simulation test based on the target test case has been completed. Therefore, the test status corresponding to the target test case can be updated to completed and displayed to the user to prompt the user to view or analyze the simulation test results. If the query result is not present, continue to wait and repeat the polling until the execution time of the simulation test exceeds the preset test timeout. At this time, if the simulation test result is still not found, the test status corresponding to the target test case can be updated to abnormal and displayed to the user to prompt the user to terminate the simulation test, handle the existing abnormal problems and reschedule the simulation test.

[0227] The present specification also provides a computer program product, which includes a computer program, which can be used to perform the above-mentioned Figure 1 Test methods provided.

[0228] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 Test methods provided.

[0229] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the executor of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0230] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0231] For the convenience of description, the above device is described by dividing it into various units according to its functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0232] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0233] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0234] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1A function specified in one or more boxes.

[0235] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0236] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0237] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0238] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0239] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0240] It should be understood by those skilled in the art that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0241] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0242] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0243] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A testing method, characterized in that: include: Pre-acquire system parameters of each handling device and each simulation vehicle model file; the simulation vehicle model file includes a simulation vehicle model and various vehicle parameters, the simulation vehicle models included in different simulation vehicle model files are not completely the same, and the various vehicle parameters included in different simulation vehicle model files are different; Determine each test case according to each handling equipment system parameter and each simulation vehicle model file, wherein the handling equipment system parameter in the test case at least includes a handling equipment parameter and a sensor parameter; For each test case, according to the simulation vehicle model and various vehicle parameters contained in the simulation vehicle model file in the test case, determine the initial pose of the simulation vehicle model; According to the handling equipment parameters in the test case, determine the initial posture of the simulation handling equipment model; Determining a reference relative pose of the simulated carrier model relative to the simulated carrier model according to the initial pose of the simulated carrier model and the initial pose of the simulated carrier model; Generate a simulated vehicle point cloud corresponding to the test case according to the sensor parameters in the test case and the reference relative pose; In response to a simulation test request, a target test case among the test cases is called, and a simulation test is performed on the perception algorithm to be tested according to the simulated vehicle point cloud corresponding to the target test case to obtain a simulation test result of the perception algorithm to be tested, wherein the perception algorithm to be tested is used to identify the vehicle and determine the position and posture of the vehicle relative to the handling equipment.

2. The method according to claim 1, characterized in that The system parameters of each handling equipment also include at least one of each simulation scene parameter and each posture transformation strategy; wherein the simulated sensor described by the sensor parameters is configured on the simulated handling equipment model described by the handling equipment parameters, and the posture transformation strategy is used to indicate the transformation of the posture of the simulated handling equipment model or the posture of the simulated vehicle model.

3. The method according to claim 1, characterized in that Determine each test case according to the system parameters of each handling equipment and the model files of each simulation vehicle, specifically including: Selecting at least one simulation vehicle model file from the simulation vehicle model files; Selecting at least one group of handling equipment system parameters from the handling equipment system parameters; wherein the selected group of handling equipment system parameters includes at least one group of handling equipment parameters and at least one group of sensor parameters; Generate test tasks according to the selected simulation vehicle model file and the selected handling equipment system parameters; The simulation vehicle model file and the handling equipment system parameters included in the test task are arranged and combined to obtain test cases corresponding to the test task.

4. The method according to claim 1, characterized in that Determine each test case according to the system parameters of each handling equipment and the model files of each simulation vehicle, specifically including: In response to the user's selection operation, at least one selected simulation vehicle model file is determined from the simulation vehicle model files, and at least one group of selected handling equipment system parameters is determined from the handling equipment system parameters; wherein the group of selected handling equipment system parameters includes at least one group of handling equipment parameters and at least one group of sensor parameters; Generate test cases based on the selected simulation vehicle model file and the selected handling equipment system parameters.

5. The method according to any one of claims 3 or 4, characterized in that: A set of selected handling equipment system parameters also includes at least one set of simulation scene parameters and / or at least one set of posture transformation strategies.

6. The method according to claim 1, characterized in that The handling equipment system parameters in this test case also include the posture transformation strategy; Determining a reference relative pose of the simulated carrier model relative to the simulated carrier model according to the initial pose of the simulated carrier model and the initial pose of the simulated carrier model specifically includes: When the posture transformation strategy in the test case is the first posture transformation strategy, the initial posture of the simulated handling equipment model is transformed, and the reference relative posture of the simulated vehicle model relative to the simulated handling equipment model is determined according to the transformed posture of the simulated handling equipment model and the initial posture of the simulated vehicle model; When the posture transformation strategy in the test case is the second posture transformation strategy, the initial posture of the simulated vehicle model is transformed, and the reference relative posture of the simulated vehicle model relative to the simulated handling equipment model is determined based on the initial posture of the simulated handling equipment model and the transformed posture of the simulated vehicle model.

7. The method according to claim 1, characterized in that The handling equipment system parameters in this test case also include simulation scenario parameters; Generating a simulated vehicle point cloud corresponding to the test case according to the sensor parameters in the test case and the reference relative pose, specifically including: According to the sensor parameters and simulation scene parameters in the test case, as well as the reference relative posture, a simulation vehicle point cloud corresponding to the test case is generated.

8. The method according to claim 1, characterized in that Performing a simulation test on the perception algorithm to be tested according to the simulated vehicle point cloud corresponding to the target test case to obtain a simulation test result of the perception algorithm to be tested specifically includes: The simulated vehicle point cloud corresponding to the target test case is input into the perception algorithm to be tested, and the target relative posture of the simulated vehicle model relative to the simulated handling equipment model in the target test case is identified from the simulated vehicle point cloud corresponding to the target test case through the perception algorithm to be tested; The reference relative posture of the simulated vehicle model relative to the simulated handling equipment model in the target test case is used as a reference, and the simulation test result of the perception algorithm to be tested is determined according to the difference between the target relative posture and the reference relative posture.

9. The method according to claim 1, characterized in that One test case corresponds to multiple groups of simulated vehicle point clouds, and the simulated vehicle models corresponding to different groups of simulated vehicle point clouds have different reference relative postures relative to the simulated handling equipment model.

10. The method according to claim 1, characterized in that Performing a simulation test on the perception algorithm to be tested according to the simulated vehicle point cloud corresponding to the target test case to obtain a simulation test result of the perception algorithm to be tested specifically includes: For each group of simulated vehicle point clouds corresponding to the target test case, the group of simulated vehicle point clouds is input into the perception algorithm to be tested, and the target relative posture of the simulated vehicle model corresponding to the group of simulated vehicle point clouds relative to the simulated handling equipment model is identified from the group of simulated vehicle point clouds by the perception algorithm to be tested; Taking the reference relative pose of the simulated vehicle model corresponding to the group of simulated vehicle point clouds relative to the simulated handling equipment model as a reference, and determining the simulation test result of the perception algorithm to be tested detecting the group of simulated vehicle point clouds according to the difference between the reference relative pose corresponding to the group of simulated vehicle point clouds and the target relative pose corresponding to the group of simulated vehicle point clouds; The simulation test results of each group of simulated vehicle point clouds corresponding to the target test case are detected respectively according to the perception algorithm to be tested, and the simulation test results of the perception algorithm to be tested are determined.

11. The method according to claim 1, characterized in that Different groups of simulated vehicle point clouds correspond to different test distances, where the test distance is the distance between the vehicle fork entry surface of the simulated vehicle model and the vehicle body center of the simulated handling equipment model.

12. The method according to claim 1, characterized in that The perception algorithm to be tested includes multiple perception algorithms, and different perception algorithms detect different characteristics of the vehicle when identifying the vehicle posture.

13. The method according to claim 1, characterized in that Performing a simulation test on the perception algorithm to be tested according to the simulated vehicle point cloud corresponding to the target test case to obtain a simulation test result of the perception algorithm to be tested specifically includes: For each perception algorithm to be tested, the simulated vehicle point cloud corresponding to the target test case is input into the perception algorithm to be tested, and the perception algorithm to be tested detects the characteristics of the vehicle and identifies the relative pose of the simulated vehicle model in the target test case relative to the simulated handling equipment model from the simulated vehicle point cloud corresponding to the target test case as the target relative pose output by the perception algorithm to be tested; The reference relative posture of the simulated vehicle model relative to the simulated handling equipment model in the target test case is used as a reference, and the simulation test result of the perception algorithm to be tested is determined according to the difference between the target relative posture output by the perception algorithm to be tested and the reference relative posture.

14. The method according to claim 1, wherein: The method further comprises: Pre-acquire a real vehicle point cloud corresponding to the target test case; In response to a real test request, the perception algorithm to be tested is tested according to the real vehicle point cloud corresponding to the target test case to obtain a real test result of the perception algorithm to be tested; A comprehensive test result of the perception algorithm to be tested is determined according to the simulation test result and the actual test result.

15. The method according to claim 1, wherein: Get the model files of each simulation vehicle in advance, including: Pre-receive each simulated vehicle model and the corresponding size of each simulated vehicle model input by the user; Determining parameters of each vehicle according to the sizes corresponding to each simulation vehicle model; Generate each simulation vehicle model file according to each simulation vehicle model and each vehicle parameter.

16. The method according to claim 1, wherein: The method further comprises: Performing statistics on simulation test results of the perception algorithm to be tested to determine the accuracy of the perception algorithm to be tested in detecting the pose of the simulated vehicle model corresponding to the target test case; The accuracy of the posture of the simulated vehicle model corresponding to the target test case is detected according to the determined perception algorithm to be tested, and a test report corresponding to the perception algorithm to be tested is generated.

17. The method according to claim 1, wherein: Performing a simulation test on the perception algorithm to be tested according to the simulated vehicle point cloud corresponding to the target test case to obtain a simulation test result of the perception algorithm to be tested specifically includes: Get the preset query cycle and the preset test timeout; Whenever the query cycle is reached, query whether there is a simulation test result of the perception algorithm to be tested obtained by executing the target test case; if so, update the test status of the target test case to completed, and display the completed test status of the target test case to the user; if not, continue to wait; When the preset test timeout is reached, and the simulation test result of the perception algorithm to be tested obtained by executing the target test case does not exist, the test status of the target test case is updated to abnormal, and the abnormal test status of the target test case is displayed to the user.

18. A computer program product, characterized in that The method comprises a computer program, which, when executed by a processor, implements the method described in any one of claims 1 to 17.

19. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 17 is implemented.

Images