Model evaluation method and device, electronic equipment, storage medium and vehicle

Abstract

The invention provides a model evaluation method, and relates to the technical field of data processing, in particular to the technical field of automatic driving and model evaluation. The specific implementation scheme is as follows: acquiring obstacle recognition data corresponding to a to-be-evaluated image; the to-be-evaluated image is an image acquired by the test vehicle under the condition that at least one target obstacle is placed at a preset distance in at least one preset direction of the test vehicle; the obstacle identification data at least comprises an obstacle identifier and an obstacle coordinate of an identified target obstacle which are identified by using a to-be-evaluated model to identify the to-be-evaluated image; the target obstacle is in a static state or the target obstacle and the test vehicle are in a moving state and are kept relatively static; acquiring an obstacle coordinate true value corresponding to the obstacle identifier; and obtaining an evaluation result of the to-be-evaluated model at least according to the obstacle coordinate truth value and the obstacle coordinate.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, in particular to the field of autonomous driving and model evaluation technology, and specifically to a model evaluation method, device, electronic device, storage medium, and vehicle. Background Art

[0002] In the development of autonomous driving systems, obstacle perception and type recognition is one of its core functions. In order to ensure the safety and reliability of autonomous driving systems in practical applications, it is particularly important to quickly and accurately evaluate the precision of static obstacle perception and type recognition.

[0003] The traditional method first requires a large number of actual road tests to collect image or video data containing static obstacles. The collected data is annotated in detail, and the obstacle perception evaluation results are obtained by comparing the annotated information with the actual recognition data. Summary of the invention

[0004] The present disclosure provides a model evaluation method, device, electronic device, storage medium, and vehicle for solving at least one of the above-mentioned technical problems.

[0005] According to a first aspect of the present disclosure, a data processing file transmission method is provided, the method comprising:

[0006] Obtain obstacle recognition data corresponding to the image to be evaluated; the image to be evaluated is an image acquired by the test vehicle when at least one target obstacle is placed at a preset distance in at least one preset direction of the test vehicle; the obstacle recognition data at least includes an obstacle identifier and obstacle coordinates of the target obstacle identified by using the model to be evaluated to identify the image to be evaluated; the target obstacle is in a stationary state or the target obstacle and the test vehicle are in a moving state and remain relatively stationary;

[0007] Obtaining the true value of the obstacle coordinates corresponding to the obstacle identifier;

[0008] An evaluation result of the model to be evaluated is obtained at least according to the true value of the obstacle coordinates and the obstacle coordinates.

[0009] According to a second aspect of the present disclosure, there is provided a data processing device, the device comprising:

[0010] A data acquisition module, used to acquire obstacle recognition data corresponding to an image to be evaluated; the image to be evaluated is an image acquired by the test vehicle when at least one target obstacle is placed at a preset distance in at least one preset direction of the test vehicle; the obstacle recognition data at least includes an obstacle identifier and obstacle coordinates of the target obstacle identified by using the model to be evaluated to identify the image to be evaluated; the target obstacle is in a stationary state or the target obstacle and the test vehicle are in a moving state and remain relatively stationary;

[0011] An initialization module, used to obtain the true value of the obstacle coordinates corresponding to the obstacle identifier;

[0012] A data processing module is used to obtain an evaluation result of the model to be evaluated based on at least the true value of the obstacle coordinates and the obstacle coordinates.

[0013] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above method.

[0017] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the above method.

[0018] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the above method when executed by a processor.

[0019] According to a sixth aspect of the present disclosure, there is provided an autonomous driving vehicle, comprising the electronic device provided above.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0022] Figure 1 is a flow chart of a model evaluation method provided by an embodiment of the present disclosure;

[0023] Figure 2 It is a flowchart of some steps of another model evaluation method provided by an embodiment of the present disclosure;

[0024] Figure 3 It is a flowchart of some steps of another model evaluation method provided by an embodiment of the present disclosure;

[0025] Figure 4 It is a flowchart of some steps of another model evaluation method provided by an embodiment of the present disclosure;

[0026] Figure 5 It is a flowchart of some steps of another model evaluation method provided by an embodiment of the present disclosure;

[0027] Figure 6 It is a flowchart of some steps of another model evaluation method provided by an embodiment of the present disclosure;

[0028] Figure 7 It is a flowchart of some steps of another model evaluation method provided by an embodiment of the present disclosure;

[0029] Figure 8 It is a flowchart of some steps of another model evaluation method provided by an embodiment of the present disclosure;

[0030] Fig. 9 It is a flowchart of some steps of another model evaluation method provided by an embodiment of the present disclosure;

[0031] Fig.10 is a structural schematic diagram of a data processing device provided by an embodiment of the present disclosure;

[0032] Fig.11 The block diagram is a block diagram of an electronic device for implementing the method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0034] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0035] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0036] The terminology used herein is for describing particular embodiments only and is not intended to limit the present disclosure.As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0037] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined as such herein.

[0038] The data processing method according to the present disclosure can be executed by an electronic device such as a terminal device or a server, and the terminal device can be a vehicle-mounted device, a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The method can be implemented by a processor calling a computer-readable program instruction stored in a memory. Alternatively, the data processing method provided by the present disclosure can be executed by a server.

[0039] In the first embodiment disclosed, see Figure 1 , Figure 1 A flowchart of a file transmission method provided by the first embodiment of the present disclosure is shown, and the method includes:

[0040] S101, obtaining obstacle recognition data corresponding to an image to be evaluated; the image to be evaluated is an image obtained by a test vehicle when at least one target obstacle is placed at a preset distance in at least one preset direction of the test vehicle; the obstacle recognition data at least includes an obstacle identifier and obstacle coordinates of the target obstacle identified by using the model to be evaluated to identify the image to be evaluated; the target obstacle is stationary or the target obstacle and the test vehicle are in motion and remain relatively stationary;

[0041] S102: Obtain the true value of the obstacle coordinates corresponding to the obstacle identifier.

[0042] S103: Obtain an evaluation result of the model to be evaluated at least according to the true value of the obstacle coordinates and the obstacle coordinates.

[0043] For example, the image to be evaluated may be an image of a target obstacle acquired from the test vehicle after one or more target obstacles are placed around the test vehicle in different preset directions and at different preset distances.

[0044] It should be emphasized that when acquiring the image to be evaluated, the target obstacle can be in a stationary state. The target obstacle and the test vehicle can also be in a moving state, but during the movement, the target obstacle and the test vehicle remain relatively stationary, that is, the moving direction and speed of the target obstacle and the test vehicle are completely consistent.

[0045] Among them, the target obstacles can be common obstacles such as cone barrels, pedestrians, non-motor vehicles, motor vehicles, etc. In a test process, multiple target obstacles placed can be target obstacles of the same type.

[0046] In some possible implementations, the preset direction may include but is not limited to the front, left front, right front, left, right, rear, left rear, right rear, and any other direction that the test vehicle can obtain.

[0047] The preset distance may include but is not limited to 5m, 10m, 15m, 20m, 25m, 30m, 35m, 40m, 45m, etc., which are obstacle distances that may affect the driving of the test vehicle during actual driving.

[0048] In some possible implementations, the test vehicle may be an autonomous vehicle, and the image to be evaluated may be an image frame of a video captured by a camera on the autonomous vehicle, wherein the camera may be located at any position of the autonomous vehicle as long as the target obstacle can be captured.

[0049] In some possible implementations, a video may include multiple image frames, and each image frame may be used as an image to be evaluated and processed using the model evaluation method provided in the embodiments of the present disclosure.

[0050] For multiple image frames, the image frames can be processed according to the frame sequence corresponding to the image frames using the model evaluation method provided in the embodiment of the present disclosure. After each image frame is processed, a real-time evaluation result of the model to be evaluated can be obtained based on the image frame and the image frame that is before the image frame in the frame sequence. After processing the last image frame, the final evaluation result of the model to be evaluated is obtained.

[0051] In some possible implementations, the obstacle recognition data corresponding to the image to be evaluated may be data output by the model to be evaluated when the image to be evaluated is input into the model to be evaluated.

[0052] In some possible implementations, when there are multiple images to be evaluated, the images to be evaluated may be input into the model to be evaluated according to the frame sequence corresponding to the images to be evaluated in the video, and the obstacle recognition data output by the model to be evaluated may be processed in sequence.

[0053] The model to be evaluated is used to identify the image to be evaluated, identify obstacles in the image to be evaluated, mark the obstacles, generate an obstacle ID (identifier) ​​to distinguish the obstacle from other elements in the image, and determine the obstacle coordinates of the obstacle (specifically, the image coordinates of the obstacle in the image to be evaluated).

[0054] The obstacle identifications obtained from different images to be evaluated may be consistent, and the obstacle identifications of the same target obstacle identified in different images to be evaluated are theoretically consistent. However, the obstacle identifications corresponding to multiple target obstacles detected in the same image are inconsistent.

[0055] In some possible implementations, the model to be evaluated may also generate other information, which may also be used as obstacle identification data.

[0056] In some possible implementations, in step S101 , obstacle data corresponding to the image to be evaluated is obtained, which may be obstacle identification data output by the model to be evaluated, through communication with the model to be evaluated.

[0057] In some possible implementations, when the target obstacle is a stationary target obstacle, since the coordinates output by the model to be evaluated may be coordinates in image coordinates, the coordinates in image coordinates can be converted into coordinates in a coordinate system such as the world coordinate system by obtaining the camera parameters of the camera device of the image to be evaluated.

[0058] In some possible implementations, in step S102 and step S103, after the obstacle identification data is obtained, an obstacle identifier is obtained from the obstacle identification data, and it is determined whether the true value of the obstacle coordinates corresponding to the obstacle identifier is empty.

[0059] In some possible implementations, if the true value of the obstacle coordinates corresponding to the obstacle identification is empty, there is a lack of evaluation criteria for evaluating the obstacle coordinates detected by the model to be evaluated, and the model to be evaluated cannot be evaluated. Therefore, it is necessary to evaluate the model to be evaluated based on the true value of the coordinates and the obtained obstacle coordinates when the true value of the obstacle coordinates corresponding to the obstacle identification is not empty.

[0060] In the model evaluation method provided in the embodiment of the present disclosure, the true value of the obstacle coordinates can be determined according to the obstacle coordinates output by the model to be evaluated when the target obstacle is detected for the first time, or the true value of the obstacle coordinates can be determined according to other methods.

[0061] In some possible implementations, when the true value of the obstacle coordinates is not empty, the obstacle coordinates are compared with the true value of the obstacle coordinates, and the evaluation result of the model to be evaluated is determined according to the error between the obstacle coordinates and the true value of the obstacle coordinates.

[0062] When the test vehicle and the target obstacle are relatively stationary, the coordinates of the target obstacle relative to the test vehicle are obviously fixed. Therefore, in the coordinate system with the test vehicle as the coordinate center, the coordinates of the target obstacle do not change. If the obstacle coordinates and the true value of the obstacle coordinates are converted to the coordinate system with the test vehicle as the coordinate center, the smaller the error between the converted obstacle coordinates and the true value of the obstacle coordinates, it means that the coordinates of the target obstacles detected by the model to be evaluated are within the preset coordinate range, which is consistent with the fixed coordinates of the target obstacle relative to the test vehicle, and the detection of the model to be evaluated is accurate.

[0063] On the contrary, the greater the error between the obstacle coordinates and the true value of the obstacle coordinates, the greater the error is, indicating that the coordinates of the target obstacle detected by the model to be evaluated are not all within the preset coordinate range, which is inconsistent with the fixed coordinates of the target obstacle relative to the test vehicle, and the detection of the model to be evaluated is inaccurate.

[0064] When the target obstacle is stationary, the coordinates of the target obstacle in objective coordinate systems such as the world coordinate system and high-precision map coordinates will not change. No matter how the test vehicle moves, the obstacle coordinates of the target obstacle obtained through the image to be evaluated should not change when converted to the coordinates in objective coordinate systems such as the world coordinate system. The smaller the error between the obstacle coordinates after coordinate conversion and the true value of the obstacle coordinates, the smaller the coordinates of the target obstacle obtained through the model to be evaluated are within the preset coordinate range, which is consistent with the target obstacle being stationary. Therefore, the detection of the model to be evaluated is accurate.

[0065] On the contrary, the greater the error between the obstacle coordinates and the true value of the obstacle coordinates, the greater the error is, indicating that the coordinates of the target obstacle detected by the model to be evaluated are not all within the preset coordinate range, which is inconsistent with the target obstacle being stationary. Therefore, the detection of the model to be evaluated is inaccurate.

[0066] In the method provided by the embodiment of the present disclosure, target obstacles are set in different directions and at different distances, obstacle recognition data of the image to be evaluated is obtained, and the obstacle data is compared with the true value of the obstacle coordinates to evaluate the model to be evaluated for obtaining the obstacle recognition data, thereby obtaining the recognition status of the model to be evaluated for obstacles in different directions and at different distances.

[0067] At the same time, the method provided by the embodiment of the present disclosure can be applied to obstacle assessment scenarios of different types and numbers, and has strong scalability.

[0068] The model evaluation method provided by the embodiment of the present disclosure is specifically introduced below.

[0069] As mentioned above, no matter the target obstacle is stationary or the target obstacle and the test vehicle are in motion and remain relatively stationary, it is necessary to convert the true value of the obstacle coordinates and the obstacle coordinates into the corresponding coordinate system and then compare them to obtain the evaluation result of the model to be evaluated.

[0070] Figure 2 The figure shows a flow chart of an implementation method of performing coordinate transformation on the obstacle coordinates and the true value of the obstacle coordinates to obtain the evaluation result of the model to be evaluated when the target obstacle is in a stationary state. Figure 2 As shown, it may include steps S201, S202, and S203.

[0071] S201, obtaining coordinate conversion parameters of obstacle coordinates relative to the high-precision map;

[0072] S202, performing coordinate transformation on the true value of the coordinates of the obstacle and the coordinates of the obstacle according to the coordinate transformation parameters;

[0073] S203: Obtain an evaluation result of the model to be evaluated at least according to the true value of the obstacle coordinates after the coordinate transformation and the obstacle coordinates after the coordinate transformation.

[0074] In step S201, when the obstacle coordinates output by the model to be evaluated are the image coordinates of the image to be evaluated, the coordinate conversion parameters of the obstacle coordinates relative to the high-precision map can be obtained based on the camera parameters of the camera device that obtains the image to be evaluated and the parameters of the high-precision map.

[0075] In step S202, coordinate transformation is performed on the true value of the coordinates of the obstacle and the coordinates of the obstacle according to the acquired coordinate transformation parameters.

[0076] It should be emphasized that the embodiments of the present disclosure do not limit the methods for obtaining coordinate conversion parameters and performing coordinate conversion, and any method that can achieve coordinate conversion is within the protection scope of the embodiments of the present disclosure.

[0077] In step S203, the specific implementation method of obtaining the evaluation result of the model to be evaluated according to the true value of the obstacle coordinates after coordinate transformation and the obstacle coordinates after coordinate transformation is as described in the following implementation method, which will not be repeated here.

[0078] In this implementation, a high-precision map (HD map) coordinate system is selected as the objective coordinate system. After the true value of the obstacle coordinate and the obstacle coordinate are converted to the high-precision map coordinate system, the evaluation result of the model to be evaluated is obtained based on the true value of the obstacle coordinate after the coordinate conversion and the obstacle coordinate after the coordinate conversion.

[0079] A high-precision map (HD map) is a high-precision map used for autonomous driving. As a priori data for autonomous driving, it expresses map elements such as roads, lanes, roadside traffic signs and ground markings. The map accuracy can be down to the centimeter level.

[0080] Therefore, by converting the true value of the obstacle coordinates and the obstacle coordinates into a high-precision map coordinate system, the coordinate values ​​with centimeter accuracy can also be obtained, and then the comparison accuracy between the true value of the obstacle coordinates and the obstacle coordinates can be improved to centimeter accuracy, further improving the accuracy of the evaluation results of the evaluation model.

[0081] Figure 3 A flow chart of an implementation method of performing coordinate transformation on obstacle coordinates and obstacle coordinate true values ​​to obtain evaluation results of a model to be evaluated is shown when a target obstacle and a test vehicle are in motion and remain relatively stationary.

[0082] S301, obtaining coordinate conversion parameters of the obstacle coordinates relative to the test vehicle;

[0083] S302, performing coordinate transformation on the true value of the coordinates of the obstacle and the coordinates of the obstacle according to the coordinate transformation parameters;

[0084] S303: Obtain an evaluation result of the model to be evaluated at least according to the true value of the obstacle coordinates after the coordinate transformation and the obstacle coordinates after the coordinate transformation.

[0085] In step S301, when the obstacle coordinates output by the model to be evaluated are image coordinates of the image to be evaluated, coordinate conversion parameters of the obstacle coordinates relative to the test vehicle can be obtained according to camera parameters of the camera device that obtains the image to be evaluated and parameters of the test vehicle.

[0086] In step S302, coordinate transformation is performed on the true value of the coordinates of the obstacle and the coordinates of the obstacle according to the acquired coordinate transformation parameters.

[0087] It should be emphasized that the embodiments of the present disclosure do not limit the methods for obtaining coordinate conversion parameters and performing coordinate conversion, and any method that can achieve coordinate conversion is within the protection scope of the embodiments of the present disclosure.

[0088] In step S303, the specific implementation method of obtaining the evaluation result of the model to be evaluated according to the true value of the obstacle coordinates after coordinate transformation and the obstacle coordinates after coordinate transformation is as described in the following implementation method, which will not be repeated here.

[0089] In this implementation, a coordinate system centered on the test vehicle is selected as the reference coordinate system. After the true value of the obstacle coordinates and the obstacle coordinates are converted to the test vehicle coordinate system, the evaluation result of the model to be evaluated is obtained based on the true value of the obstacle coordinates after the coordinate conversion and the obstacle coordinates after the coordinate conversion.

[0090] In some possible implementations, if the camera device that obtains the image to be evaluated is a camera device in a test vehicle, the coordinate conversion parameters can be obtained according to the camera parameters of the camera device and the moving direction and moving distance of the camera device in the test vehicle.

[0091] If the camera device does not move, the image coordinates can be directly used as the coordinates of the test vehicle coordinate system without the need for coordinate conversion, thereby simplifying the steps and saving computing resources.

[0092] The following specifically introduces the implementation method of obtaining the evaluation result of the model to be evaluated based on the true value of the obstacle coordinates after coordinate transformation and the obstacle coordinates after coordinate transformation.

[0093] As described above, in some possible implementations, the evaluation result of the model to be evaluated includes the recall success rate of the model to be evaluated.

[0094] Figure 4 FIG. 1 is a flow chart showing an implementation method of determining the recall success rate based on the true value of the obstacle coordinates and the obstacle coordinates when the true value of the obstacle coordinates is determined. Figure 4 As shown, it may include:

[0095] S401, obtaining a recall success value corresponding to the model to be evaluated, and updating the recall success value when the distance between the obstacle coordinates and the true value of the obstacle coordinates is less than a preset threshold;

[0096] S402: Determine the recall success rate corresponding to the model to be evaluated according to the recall success value.

[0097] In some possible implementations, in step S401, a value is required to calculate the number of successful recalls, and then the recall success rate is calculated based on the number of successful recalls.

[0098] The recall success value obtained according to the obstacle data corresponding to the previous image to be evaluated can be obtained as the initial recall success value in the calculation process, and then the distance between the obstacle coordinates and the true value of the obstacle coordinates is calculated. When the calculated distance is less than the preset threshold, the recall success value is updated.

[0099] The method of calculating the distance and the specific distance calculated are not limited in the embodiments of the present disclosure.

[0100] In some possible implementations, in step S402, the real-time recall success rate corresponding to the model to be evaluated may be determined according to the recall success value and the number of images to be evaluated that have been identified by the model to be evaluated at this time.

[0101] By calculating the distance between the obstacle coordinates and the true value of the obstacle coordinates, obstacles that are far away from the true value of the obstacle coordinates identified by the evaluation model are considered to have failed recall, that is, obstacles with large coordinate determination errors are considered to have failed recall. Corresponding to the actual application scenario, if the coordinate recognition is far away from the actual position of the obstacle, the recognition is considered to have failed.

[0102] Therefore, the recall success rate is used to evaluate the accuracy of the detection model in locating target obstacles in different directions and at different distances. The recall rate calculated in this way is more in line with the needs of actual driving scenarios and may also ensure the application safety of the model to be evaluated.

[0103] In some possible implementations, after all the images to be evaluated are identified, the final recall success rate corresponding to the model to be evaluated can be determined according to the ratio of the final output recall success value to the number of all the images to be evaluated.

[0104] The final recall success rate obtained in this way is a stable recall success rate within a certain period of time, which is more valuable for reference to the evaluation model.

[0105] In some specific implementations, after obtaining the recall success value obtained based on the obstacle data corresponding to the previous image to be evaluated, the distance between the obstacle coordinates and the true value of the obstacle coordinates is calculated. When the calculated distance is less than a preset threshold, the obtained recall success value is added by 1 to update the recall success value.

[0106] The real-time recall success rate of the model to be evaluated is determined based on the ratio of the recall success value to the number of images to be evaluated that the model to be evaluated has identified at this time. After all images to be evaluated are involved in the calculation, the final recall success rate of the model to be evaluated is determined.

[0107] As described above, in some possible implementations, the obstacle recognition data also includes the obstacle type of the target obstacle identified by using the model to be evaluated to identify the image to be evaluated; the evaluation result of the model to be evaluated also includes the detection accuracy of the model to be evaluated.

[0108] In some possible implementations, the obstacle type may be represented in digital form.

[0109] Figure 5 A flowchart of an implementation method for calculating the detection accuracy of the model to be evaluated is shown in FIG. Figure 5The model evaluation method provided by the embodiment of the present disclosure may include:

[0110] S501, obtaining the type true value corresponding to the target obstacle;

[0111] S502: Determine the detection accuracy of the model to be evaluated according to the obstacle type and the true value of the type.

[0112] In some possible implementations, in S501, obtaining the true value of the type corresponding to the target obstacle may be directly obtaining a number corresponding to the obstacle type of the placed target obstacle.

[0113] In some possible implementations, in S502, the acquired obstacle type may be compared with the true value of the type to determine the detection accuracy of the model to be evaluated.

[0114] Since only when the obstacle is recalled can we calculate whether the obstacle detection is correct, and then determine the detection accuracy of the model to be detected. The above method can ensure that the calculated detection accuracy is the detection accuracy obtained when the obstacle is recalled.

[0115] In some possible implementations, as described above, all obstacles involved in calculating the true value of the obstacle coordinates are recalled successfully by default. In the case of a successful recall, it is necessary to determine the obstacle type of the target obstacle that has been recalled successfully. Therefore, the obstacle data used to determine the true value of the obstacle coordinates should also be involved in calculating the detection accuracy.

[0116] Figure 6 FIG. 1 is a flow chart showing a specific implementation method for determining the detection accuracy when the true value of the obstacle coordinates is determined. Figure 6 As shown, determining the detection accuracy may include:

[0117] S601, obtaining the detection accuracy value corresponding to the model to be evaluated;

[0118] S602: if the distance between the obstacle coordinates and the true value of the obstacle coordinates is less than a preset threshold and the obstacle type and the true value of the type are consistent, update the detection accuracy value;

[0119] S603: Determine the detection accuracy of the model to be evaluated according to the detection accuracy value and the recall success value.

[0120] In some possible implementations, in step S601, a value is required to calculate the number of accurate detections, and then the detection accuracy rate is calculated by the number of accurate detections.

[0121] The detection accuracy value obtained according to the obstacle data corresponding to the previous image to be evaluated may be obtained as the initial detection accuracy value in the current calculation process.

[0122] In some possible implementations, in step S602, only when the distance between the obstacle coordinates and the true value of the obstacle coordinates is less than a preset threshold value can it be determined that the obstacle recall is successful. Only when the obstacle recall is successful, it is meaningful to calculate whether the obstacle is detected correctly. Therefore, only when the distance between the obstacle coordinates and the true value of the obstacle coordinates is less than the preset threshold value, the obstacle type and the true value of the type are compared to determine whether they are consistent. If they are consistent, the detection accuracy value is updated to achieve accurate recording of the detection.

[0123] In some possible implementations, in step S603, the real-time detection accuracy rate corresponding to the model to be evaluated may be determined according to the ratio of the acquired detection accuracy value and the recall success value acquired in the above step.

[0124] The detection accuracy is used to evaluate the accuracy of the model to be detected in identifying the types of target obstacles at different directions and distances. Only when the obstacle is successfully recalled, it is meaningful to calculate whether the obstacle is detected correctly. The above method can ensure that only when the obstacle is successfully recalled, the obstacle data is used to calculate the detection accuracy.

[0125] In some specific implementations, when the distance between the obstacle coordinates and the true value of the obstacle coordinates is less than a preset threshold, the obstacle type and the true value of the type are compared to determine whether they are consistent. If they are consistent, the detection accuracy value is increased by 1 to update the detection accuracy value.

[0126] The real-time detection accuracy corresponding to the model to be evaluated can be determined according to the ratio of the obtained detection accuracy value and the recall success value obtained in the above step.

[0127] In some possible implementations, after all the images to be evaluated are recognized, the final detection accuracy rate corresponding to the model to be evaluated can be determined based on the ratio of the final output detection accuracy value to the recall success value.

[0128] The following is a detailed introduction on how to determine the true value of the obstacle coordinates.

[0129] As described above, in some possible implementations, when the true value of the obstacle coordinates corresponding to the obstacle identifier is empty, the true value of the obstacle coordinates may be acquired according to the obstacle coordinates.

[0130] For example, the true value of the obstacle coordinates can be determined based on the obstacle coordinates output by the model to be evaluated when the target obstacle is detected for the first time.

[0131] Compared with labeling the data, determining the true value of the obstacle coordinates by labeling the data does not require data preprocessing and data labeling, which greatly reduces the workload of evaluation, reduces the difficulty and cost of operation, and releases computing resources so that computing resources can be used for data comparison, thereby improving the speed of data comparison and thus improving the efficiency of evaluation.

[0132] In some specific implementations, the obstacle coordinates output by the model to be evaluated when the target obstacle is detected for the first time can be directly used as the true value of the obstacle coordinates.

[0133] On the one hand, when the model to be evaluated detects the target obstacle for the first time, the true value of the obstacle coordinates can be determined, which can ensure that the obstacle coordinates obtained later can participate in the evaluation of the model to be evaluated, and as much data as possible can be involved in the evaluation of the model to be evaluated, thereby improving the accuracy of the evaluation.

[0134] On the other hand, when the model to be evaluated detects the target obstacle for the first time, the true value of the obstacle coordinates is determined, which can ensure that the obstacle coordinates obtained later can participate in the evaluation of the model to be evaluated, and as much data as possible can be involved in the evaluation of the model to be evaluated, thereby improving the accuracy of the evaluation.

[0135] In some possible implementations, in order to reduce the impact of possible errors in the true value of the obstacle target on the evaluation result, the true value of the obstacle coordinates can also be determined based on the obstacle coordinates output by the model to be evaluated when the target obstacle is detected N times before. Where N can be an integer less than the number of image frames of the video.

[0136] Figure 7 FIG. 1 is a flow chart showing an implementation method of determining the true value of the obstacle coordinates according to the obstacle coordinates output by the model to be evaluated when the target obstacle is detected N times before. Figure 7 Determining the true value of the obstacle coordinates according to the obstacle coordinates output by the model to be evaluated when detecting the target obstacle N times before may include:

[0137] S701, using the obstacle coordinates as an obstacle coordinate reference;

[0138] S702: When the number of obstacle coordinate references reaches a preset number, determine an average value of all obstacle coordinate references as a true value of the obstacle coordinate.

[0139] In some possible implementations, in step S701, the obstacle coordinates output by the model to be evaluated for the previous N times when the model detects the target obstacle are all recorded as obstacle coordinate references.

[0140] In some possible implementations, in step S702, the preset number is N, and the number of obstacle coordinate references reaches the preset number, that is, N obstacle coordinate references are obtained, all obstacle coordinate references are averaged, and the average value is determined as the true value of the obstacle coordinate.

[0141] The obstacle coordinate references are averaged by averaging the coordinate values ​​corresponding to different coordinate axes of the obstacle coordinates, and taking the average value as the coordinate value corresponding to the true value of the obstacle coordinates on the coordinate axis.

[0142] Obtaining the true value of the obstacle coordinates by averaging can reduce the impact of the detection error in the obstacle data output by the model to be evaluated when it first detects the target obstacle on the true value of the obstacle coordinates, minimize the error between the obtained obstacle target true value and the true coordinates of the obstacle, and thus reduce the impact of the error on the evaluation results.

[0143] As mentioned above, the first time the model to be evaluated detects the target obstacle is the first time the model to be evaluated is successfully recalled. Therefore, the obstacle data used to determine the true value of the obstacle coordinates should also be involved in calculating the recall success value and the recall success rate.

[0144] Figure 8 A flowchart of an implementation method for determining the obstacle coordinate true value and using the obstacle data to calculate the recall success rate is shown. Figure 8 As shown, it may include:

[0145] S801, obtaining the recall success value corresponding to the model to be evaluated;

[0146] S802: when the recall success value corresponding to the model to be evaluated is empty, initialize the recall success value;

[0147] S803: When the recall success value corresponding to the model to be evaluated is not empty, update the recall success value.

[0148] In some possible implementations, in step S801, it is determined whether the recall success value corresponding to the model to be evaluated is empty.

[0149] In some possible implementations, in step S802, when the recall success value is empty, the recall success value is initialized.

[0150] In some possible implementations, in step S803, when the recall success value is not empty, that is, has been initialized, the recall success value is updated to record the number of successful recalls.

[0151] Fig. 9 A flowchart of an implementation method for determining the true value of the obstacle coordinates using obstacle data to calculate the recall success detection accuracy is shown, such as Fig. 9 As shown, it may include:

[0152] S901, obtaining the detection accuracy value corresponding to the model to be evaluated;

[0153] S902, when the detection accuracy value corresponding to the model to be evaluated is empty, initialize the detection accuracy value;

[0154] S903, obtaining the type true value corresponding to the target obstacle;

[0155] S904: When the obstacle type is consistent with the true value of the type, update the detection accuracy value.

[0156] In some possible implementations, a detection accuracy value is required to record the number of accurate detections, and then the detection accuracy rate is calculated by the detection accuracy value and the recall success value. This value may be defined at the beginning and defined as empty. In step S901, the value is obtained.

[0157] In some possible implementations, in step S902, when the detection accuracy value is empty, the detection accuracy value is initialized.

[0158] In some possible implementations, in step S903, obtaining the type true value corresponding to the target obstacle may be directly obtaining a number corresponding to the obstacle type of the placed target obstacle.

[0159] In some possible implementations, in step S904, if the obstacle type is consistent with the true value of the type, indicating that the detection is accurate, the detection accuracy value is updated and the number of accurate detections is recorded.

[0160] In some possible implementations, the real-time detection accuracy rate corresponding to the model to be evaluated may be determined according to the ratio of the acquired detection accuracy value and the recall success value acquired in the above step.

[0161] In some possible implementations, after all the images to be evaluated are recognized, the final detection accuracy rate corresponding to the model to be evaluated can be determined based on the ratio of the final output detection accuracy value to the recall success value.

[0162] The model evaluation method provided by the embodiment of the present disclosure is specifically introduced below with a specific example.

[0163] Specifically, a plurality of stationary target obstacles may be set at a preset direction and a preset distance from the initial position of the test vehicle. The plurality of target obstacles may be distinguishable obstacles of the same type or different types with different characteristics.

[0164] Specifically, barrel cones with different logos can be set up at 5m, 10m, 15m and 20m in front of the right side of the initial position of the test vehicle.

[0165] Then the test vehicle starts to move, and during the movement, a camera device on the test vehicle captures a plurality of continuous image frames as a plurality of images to be evaluated.

[0166] The image to be evaluated is input into the model to be evaluated to recognize the image to be evaluated, and the obstacle identification, obstacle coordinates, and obstacle type of the target obstacle in the image to be evaluated are recognized.

[0167] After obtaining the obstacle coordinates, the obstacle coordinates are converted into coordinates in a high-precision map coordinate system according to the camera parameters of the camera device.

[0168] Among them, high-precision map (HD map) is a high-precision map used for autonomous driving. As a priori data for autonomous driving, it expresses map elements such as roads, lanes, roadside traffic signs and ground signs. The map accuracy can be down to the centimeter level.

[0169] It is defined as the recall success value, the detection accuracy value, and the true value of the obstacle coordinates corresponding to all target obstacles. The recall success value is used to count the number of target obstacles that the model to be detected successfully recalls, and the detection accuracy value is used to count the number of correct detections of the target obstacles that the model to be detected successfully recalls.

[0170] For the same obstacle identifier, the obstacle coordinates corresponding to the obstacle identifier are obtained and arranged according to the frame sequence of the image frames.

[0171] The first N obstacle coordinates are averaged as the true value coordinates of the obstacle corresponding to the obstacle mark, and the recall success value is updated to N.

[0172] Starting from the N+1th obstacle coordinate, the obstacle coordinate is compared with the true value coordinate of the obstacle. If the distance between the obstacle coordinate and the true value of the obstacle coordinate is less than the preset threshold, the recall success value is increased by 1.

[0173] According to the obstacle identification, the type true value corresponding to the target obstacle is obtained, and the detection accuracy value is initialized to 0. The obstacle types corresponding to the first N obstacle coordinates are compared with the type true value corresponding to the target obstacle. If they are consistent, the detection accuracy value is increased by 1.

[0174] Starting from the N+1th obstacle coordinate, when the recall success value is +1 and the obstacle type corresponding to the obstacle coordinate is consistent with the true value of the type corresponding to the target obstacle, the detection accuracy value is +1.

[0175] After calculating all the images to be evaluated, the ratio of the recall success value to the number of all images to be evaluated is used as the recall success rate of the target obstacle corresponding to the obstacle mark of the model to be evaluated, and the ratio of the detection accuracy value to the recall success value is used as the detection accuracy rate of the target obstacle corresponding to the obstacle mark of the model to be evaluated.

[0176] Among them, the recall success rate is used to evaluate the accuracy of the model to be detected in locating target obstacles in different directions and at different distances, and the detection accuracy rate is used to evaluate the accuracy of the model to be detected in identifying the types of target obstacles in different directions and at different distances.

[0177] Based on Figure 1 The same principle, Fig.10 A data processing device 100 provided in the third embodiment of the present disclosure is shown, and the device includes:

[0178] The data acquisition module 1001 is used to acquire obstacle recognition data corresponding to the image to be evaluated; the image to be evaluated is an image acquired by the test vehicle when at least one target obstacle is placed at a preset distance in at least one preset direction of the test vehicle; the obstacle recognition data at least includes the obstacle identification and obstacle coordinates of the target obstacle identified by using the model to be evaluated to identify the image to be evaluated; the target obstacle is in a stationary state or the target obstacle and the test vehicle are in a moving state and remain relatively stationary;

[0179] Initialization module 1002, used to obtain the true value of the obstacle coordinates corresponding to the obstacle identifier;

[0180] The data processing module 1003 is used to obtain the evaluation result of the model to be evaluated according to at least the true value of the obstacle coordinates and the obstacle coordinates.

[0181] In some examples, the data processing module includes:

[0182] A parameter acquisition unit, used to acquire coordinate conversion parameters of obstacle coordinates relative to the high-precision map;

[0183] A coordinate conversion unit, used for performing coordinate conversion on the true value of the coordinates of the obstacle and the coordinates of the obstacle according to the coordinate conversion parameters;

[0184] The data processing unit is used to obtain the evaluation result of the model to be evaluated according to at least the true value of the obstacle coordinates after the coordinate transformation and the obstacle coordinates after the coordinate transformation.

[0185] In some examples, a parameter acquisition unit is used to obtain coordinate transformation parameters of the obstacle coordinates relative to the test vehicle;

[0186] A coordinate conversion unit, used for performing coordinate conversion on the true value of the coordinates of the obstacle and the coordinates of the obstacle according to the coordinate conversion parameters;

[0187] The data processing unit is used to obtain the evaluation result of the model to be evaluated according to at least the true value of the obstacle coordinates after the coordinate transformation and the obstacle coordinates after the coordinate transformation.

[0188] In some examples, the evaluation result of the model to be evaluated includes the recall success rate of the model to be evaluated;

[0189] The data processing module includes:

[0190] The data processing recall unit is used to obtain the recall success value corresponding to the model to be evaluated, and update the recall success value when the distance between the obstacle coordinates and the true value of the obstacle coordinates is less than a preset threshold; and determine the recall success rate corresponding to the model to be evaluated according to the recall success value.

[0191] In some examples, the data processing recall unit is further configured to:

[0192] The recall success rate corresponding to the model to be evaluated is determined according to the ratio of the recall success value to the number of images to be evaluated recognized by the model to be evaluated.

[0193] In some examples, the obstacle recognition data also includes the obstacle type of the target obstacle identified by using the model to be evaluated to identify the image to be evaluated; the evaluation result of the model to be evaluated also includes the detection accuracy of the model to be evaluated;

[0194] The data processing module also includes:

[0195] A type truth value unit, used to obtain the type truth value corresponding to the target obstacle;

[0196] The data processing detection unit is used to determine the detection accuracy of the model to be evaluated according to the obstacle type and the true value of the type.

[0197] In some examples, the data processing and detection unit is further configured to:

[0198] Obtain the detection accuracy value corresponding to the model to be evaluated; update the detection accuracy value when the distance between the obstacle coordinates and the true value of the obstacle coordinates is less than a preset threshold and the obstacle type and the true value of the type are consistent; determine the detection accuracy of the model to be evaluated based on the detection accuracy value and the recall success value.

[0199] In some examples, the data processing and detection unit is further configured to:

[0200] The detection accuracy of the model to be evaluated is determined according to the ratio of the detection accuracy value and the recall success value.

[0201] In some examples, the model evaluation device further includes: an initialization module;

[0202] The initialization module is used to obtain the true value of the obstacle coordinates according to the obstacle coordinates when the true value of the obstacle coordinates corresponding to the obstacle identifier is empty.

[0203] In some examples, the evaluation result of the model to be evaluated includes the recall success rate of the model to be evaluated;

[0204] The initialization module includes:

[0205] A recall value acquisition unit, used to obtain a recall success value corresponding to the model to be evaluated;

[0206] A recall initialization unit, used to initialize the recall success value when the recall success value corresponding to the model to be evaluated is empty;

[0207] The recall processing unit is initialized to update the recall success value when the recall success value corresponding to the model to be evaluated is not empty.

[0208] In some examples, the obstacle recognition data also includes the obstacle type of the target obstacle identified by using the model to be evaluated to identify the image to be evaluated; the evaluation result of the model to be evaluated also includes the detection accuracy of the model to be evaluated;

[0209] The initialization module also includes:

[0210] A detection value acquisition unit, used to obtain the detection accuracy value corresponding to the model to be evaluated;

[0211] A detection initialization unit, used for initializing the detection accuracy value when the detection accuracy value corresponding to the model to be evaluated is empty;

[0212] A type truth value unit, used to obtain the type truth value corresponding to the target obstacle;

[0213] The detection processing unit is initialized to update the detection accuracy value when the obstacle type is consistent with the true value of the type.

[0214] In some examples, the initialization module includes:

[0215] The averaged coordinate calculation unit is used to use the obstacle coordinates as obstacle coordinate references; when the number of obstacle coordinate references reaches a preset number, the average value of all obstacle coordinate references is determined as the true value of the obstacle coordinates.

[0216] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0217] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, a computer program product and an autonomous driving vehicle.

[0218] The computer instructions of the non-transitory computer-readable storage medium storing computer instructions are used to make the computer execute the above method.

[0219] A computer program product comprises a computer program, which implements the above method when executed by a processor.

[0220] An autonomous vehicle comprising the electronic equipment provided above.

[0221] Fig.11 A schematic block diagram of an example electronic device 1100 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0222] like Fig.11 As shown, the device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit into a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the device 1100 can also be stored. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0223] A number of components in the device 1100 are connected to the I / O interface 1105, including: an input unit 1106, such as a keyboard, a mouse, etc.; an output unit 1107, such as various types of displays, speakers, etc.; a storage unit 1108, such as a disk, an optical disk, etc.; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1109 allows the device 1100 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0224] The computing unit 1101 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1101 performs the various methods and processes described above, such as the model evaluation method. For example, in some embodiments, the model evaluation method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1100 via the ROM 1102 and / or the communication unit 1109. When the computer program is loaded into the RAM 1103 and executed by the computing unit 1101, one or more steps of the model evaluation method described above may be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured as a model evaluation method in any other appropriate manner (e.g., by means of firmware).

[0225] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0226] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0227] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0228] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0229] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0230] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0231] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0232] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A model evaluation method, wherein: The method comprises: Obtain obstacle recognition data corresponding to the image to be evaluated; the image to be evaluated is an image acquired by the test vehicle when at least one target obstacle is placed at a preset distance in at least one preset direction of the test vehicle; the obstacle recognition data at least includes an obstacle identifier and obstacle coordinates of the target obstacle identified by using the model to be evaluated to identify the image to be evaluated; the target obstacle is in a stationary state or the target obstacle and the test vehicle are in a moving state and remain relatively stationary; Obtaining the true value of the obstacle coordinates corresponding to the obstacle identifier; An evaluation result of the model to be evaluated is obtained at least according to the true value of the obstacle coordinates and the obstacle coordinates.

2. The method according to claim 1, wherein: When the target obstacle is in a stationary state, obtaining the evaluation result of the model to be evaluated at least according to the true value of the obstacle coordinates and the obstacle coordinates includes: Obtaining coordinate conversion parameters of the obstacle coordinates relative to the high-precision map; According to the coordinate transformation parameters, coordinate transformation is performed on the true value of the coordinates of the obstacle and the coordinates of the obstacle; An evaluation result of the model to be evaluated is obtained at least according to the true value of the obstacle coordinates after coordinate transformation and the obstacle coordinates after coordinate transformation.

3. The method according to claim 1, wherein: In a case where the target obstacle and the test vehicle are in motion and remain relatively stationary, obtaining the evaluation result of the model to be evaluated at least according to the true value of the obstacle coordinates and the obstacle coordinates includes: Obtaining coordinate transformation parameters of the obstacle coordinates relative to the test vehicle; According to the coordinate transformation parameters, coordinate transformation is performed on the true value of the coordinates of the obstacle and the coordinates of the obstacle; An evaluation result of the model to be evaluated is obtained at least according to the true value of the obstacle coordinates after coordinate transformation and the obstacle coordinates after coordinate transformation.

4. The method according to claim 2 or 3, wherein: The evaluation result of the model to be evaluated includes the recall success rate of the model to be evaluated; The obtaining the evaluation result of the model to be evaluated at least according to the true value of the obstacle coordinates after coordinate transformation and the obstacle coordinates after coordinate transformation, comprises: Acquire a recall success value corresponding to the model to be evaluated, and update the recall success value when the distance between the obstacle coordinates and the true value of the obstacle coordinates is less than a preset threshold; The recall success rate corresponding to the model to be evaluated is determined according to the recall success value.

5. The method according to claim 4, wherein: Determining the recall success rate corresponding to the model to be evaluated according to the recall success value includes: The recall success rate corresponding to the model to be evaluated is determined according to the ratio of the recall success value to the number of the images to be evaluated recognized by the model to be evaluated.

6. The method according to claim 4, wherein: The obstacle identification data also includes the obstacle type of the target obstacle identified by using the model to be evaluated to identify the image to be evaluated; The evaluation result of the model to be evaluated also includes the detection accuracy of the model to be evaluated; the method also includes: Obtaining a true value of the type corresponding to the target obstacle; The detection accuracy of the model to be evaluated is determined according to the obstacle type and the true value of the type.

7. The method according to claim 6, wherein: The step of determining the detection accuracy of the model to be evaluated according to the obstacle type and the true value of the type includes: Obtaining the detection accuracy value corresponding to the model to be evaluated; When the distance between the obstacle coordinates and the true value of the obstacle coordinates is less than a preset threshold and the obstacle type and the true value of the type are consistent, updating the detection accuracy value; The detection accuracy of the model to be evaluated is determined according to the detection accuracy value and the recall success value.

8. The method according to claim 7, wherein: The step of determining the detection accuracy of the model to be evaluated according to the detection accuracy value and the recall success value includes: The detection accuracy of the model to be evaluated is determined according to the ratio of the detection accuracy value to the recall success value.

9. The method according to any one of claims 1 to 8, wherein: The obtaining the true value of the obstacle coordinate corresponding to the obstacle identifier includes: When the true value of the obstacle coordinates corresponding to the obstacle identifier is empty, the true value of the obstacle coordinates is acquired according to the obstacle coordinates.

10. The method according to claim 9, wherein: The evaluation result of the model to be evaluated includes the recall success rate of the model to be evaluated; When the true value of the obstacle coordinates corresponding to the obstacle identifier is empty, acquiring the true value of the obstacle coordinates corresponding to the obstacle identifier according to the obstacle coordinates includes: Obtaining a recall success value corresponding to the model to be evaluated; When the recall success value corresponding to the model to be evaluated is empty, initializing the recall success value; When the recall success value corresponding to the model to be evaluated is not empty, the recall success value is updated.

11. The method according to claim 10, wherein: The obstacle identification data also includes the obstacle type of the target obstacle identified by using the model to be evaluated to identify the image to be evaluated; The evaluation result of the model to be evaluated also includes the detection accuracy of the model to be evaluated; The method further comprises: Obtaining the detection accuracy value corresponding to the model to be evaluated; When the detection accuracy value corresponding to the model to be evaluated is empty, initializing the detection accuracy value; Obtaining a true value of the type corresponding to the target obstacle; When the obstacle type is consistent with the true value of the type, the detection accuracy value is updated.

12. The method according to claim 9, wherein: When the true value of the obstacle coordinates corresponding to the obstacle identifier is empty, acquiring the true value of the obstacle coordinates corresponding to the obstacle identifier according to the obstacle coordinates includes: Using the obstacle coordinates as an obstacle coordinate reference; When the number of the obstacle coordinate references reaches a preset number, an average value of all the obstacle coordinate references is determined as the true value of the obstacle coordinate.

13. A model evaluation device, wherein: The device comprises: A data acquisition module, used to acquire obstacle recognition data corresponding to an image to be evaluated; the image to be evaluated is an image acquired by the test vehicle when at least one target obstacle is placed at a preset distance in at least one preset direction of the test vehicle; the obstacle recognition data at least includes an obstacle identifier and obstacle coordinates of the target obstacle identified by using the model to be evaluated to identify the image to be evaluated; the target obstacle is in a stationary state or the target obstacle and the test vehicle are in a moving state and remain relatively stationary; An initialization module, used to obtain the true value of the obstacle coordinates corresponding to the obstacle identifier; A data processing module is used to obtain an evaluation result of the model to be evaluated based on at least the true value of the obstacle coordinates and the obstacle coordinates.

14. The device according to claim 13, wherein: The data processing module comprises: A parameter acquisition unit, used to acquire coordinate conversion parameters of the obstacle coordinates relative to the high-precision map; A coordinate conversion unit, used for performing coordinate conversion on the true value of the coordinates of the obstacle and the coordinates of the obstacle according to the coordinate conversion parameters; A data processing unit is used to obtain an evaluation result of the model to be evaluated based on at least the true value of the obstacle coordinates after coordinate transformation and the obstacle coordinates after coordinate transformation.

15. The device according to claim 13, wherein: The data processing module comprises: A parameter acquisition unit, used to acquire coordinate conversion parameters of the obstacle coordinates relative to the test vehicle; A coordinate conversion unit, used for performing coordinate conversion on the true value of the coordinates of the obstacle and the coordinates of the obstacle according to the coordinate conversion parameters; A data processing unit is used to obtain an evaluation result of the model to be evaluated based on at least the true value of the obstacle coordinates after coordinate transformation and the obstacle coordinates after coordinate transformation.

16. The device according to claim 14 or 15, wherein: The evaluation result of the model to be evaluated includes the recall success rate of the model to be evaluated; The data processing module comprises: A data processing recall unit, used to obtain a recall success value corresponding to the model to be evaluated, and update the recall success value when the distance between the obstacle coordinates and the true value of the obstacle coordinates is less than a preset threshold; The recall success rate corresponding to the model to be evaluated is determined according to the recall success value.

17. The device according to claim 16, wherein: The data processing and recalling unit is also used for: The recall success rate corresponding to the model to be evaluated is determined according to the ratio of the recall success value to the number of the images to be evaluated recognized by the model to be evaluated.

18. The device according to claim 16, wherein: The obstacle identification data also includes the obstacle type of the target obstacle identified by using the model to be evaluated to identify the image to be evaluated; The evaluation result of the model to be evaluated also includes the detection accuracy of the model to be evaluated; The data processing module also includes: A type truth value unit, used to obtain a type truth value corresponding to the target obstacle; A data processing and detection unit is used to determine the detection accuracy of the model to be evaluated according to the obstacle type and the true value of the type.

19. The device according to claim 18, wherein: The data processing and detection unit is also used for: Acquire the detection accuracy value corresponding to the model to be evaluated; and update the detection accuracy value when the distance between the obstacle coordinates and the true value of the obstacle coordinates is less than a preset threshold and the obstacle type and the true value of the type are consistent; The detection accuracy of the model to be evaluated is determined according to the detection accuracy value and the recall success value.

20. The device according to claim 19, wherein The data processing and detection unit is also used for: The detection accuracy of the model to be evaluated is determined according to the ratio of the detection accuracy value to the recall success value.

21. The device according to any one of claims 13 to 20, wherein: The model evaluation device also includes: an initialization module; The initialization module is used for obtaining the obstacle coordinate true value according to the obstacle coordinate when the obstacle coordinate true value corresponding to the obstacle identifier is empty.

22. The device according to claim 21, wherein The evaluation result of the model to be evaluated includes the recall success rate of the model to be evaluated; The initialization module includes: A recall value acquisition unit, used to acquire a recall success value corresponding to the model to be evaluated; A recall initialization unit, used for initializing the recall success value when the recall success value corresponding to the model to be evaluated is empty; The recall processing unit is initialized to update the recall success value when the recall success value corresponding to the model to be evaluated is not empty.

23. The device according to claim 22, wherein: The obstacle identification data also includes the obstacle type of the target obstacle identified by using the model to be evaluated to identify the image to be evaluated; The evaluation result of the model to be evaluated also includes the detection accuracy of the model to be evaluated; The initialization module also includes: A detection value acquisition unit, used to obtain the detection accuracy value corresponding to the model to be evaluated; A detection initialization unit, configured to initialize the detection accuracy value when the detection accuracy value corresponding to the model to be evaluated is empty; A type truth value unit, used to obtain a type truth value corresponding to the target obstacle; The detection processing unit is initialized to update the detection accuracy value when the obstacle type is consistent with the true value of the type.

24. The device according to claim 21, wherein The initialization module includes: The averaged coordinate calculation unit is used to use the obstacle coordinate as an obstacle coordinate reference; when the number of the obstacle coordinate references reaches a preset number, determine the obstacle coordinate true value according to the average value of all the obstacle coordinate references.

25. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.

26. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-12.

27. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 12.

28. An autonomous driving vehicle comprising the electronic device described in claim 25.

Images