Target detection algorithm online evaluation method, system, electronic device and storage medium

Abstract

An embodiment of the present invention provides an online evaluation method and system for a visual 3D target detection algorithm, wherein the method comprises: obtaining a sample data set of the visual 3D target detection algorithm, the sample data set comprising original sample images with a frequency greater than 2 Hz and carrying annotation information; inputting the first original sample image of the sample data set into the visual 3D target detection algorithm to output a detection result; and performing an index evaluation on the detection result and the annotation information of the second original sample image at the current moment in the sample data set. The frequency of the original sample image of the embodiment of the present invention is higher than the frequency of the sample data in the data set nuScenes, thereby increasing the sample frequency of the visual 3D target detection algorithm. When the running time of the visual 3D target detection algorithm is stable, increasing the sample frequency is equivalent to reducing the displacement deviation of the target object between the two sample images, and the index evaluation of the visual 3D target detection algorithm is more in line with the real-time requirements.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an online evaluation method for a visual 3D target detection algorithm, an online evaluation system for a visual 3D target detection algorithm, an electronic device, and a computer-readable storage medium. Background Art

[0002] In recent years, vision-centric perception algorithms have flourished in various autonomous driving tasks, including 3D detection, semantic map construction, motion prediction, and depth estimation. However, the latency of vision-centric perception algorithms is too high for practical deployment (e.g., the running time of most camera-based 3D detectors is greater than 300ms). In order to bridge the gap between ideal research and real-world applications, it is necessary to quantify the trade-off between performance and efficiency.

[0003] However, existing camera-based 3D detection algorithm performance evaluation schemes are mainly evaluated on the open source dataset nuScenes. However, the dataset nuScenes only annotates 2 Hz sample data, which cannot meet the requirements of real-time evaluation. Summary of the invention

[0004] In view of the above problems, embodiments of the present invention are proposed to provide a method for online evaluation of a visual 3D target detection algorithm and a corresponding online evaluation system for a visual 3D target detection algorithm that overcome the above problems or at least partially solve the above problems.

[0005] In order to solve the above problems, an embodiment of the present invention discloses an online evaluation method for a visual 3D target detection algorithm, the method comprising: obtaining a sample data set of the visual 3D target detection algorithm to be evaluated, the sample data set comprising multiple original sample images with preset frequencies and carrying annotation information, the preset frequency being greater than 2 Hz; inputting a first original sample image in the sample data set into the visual 3D target detection algorithm, and outputting a detection result; performing an index evaluation on the detection result and the annotation information of a second original sample image in the sample data set; wherein the second original sample image is the original sample image corresponding to the moment of outputting the detection result.

[0006] Optionally, obtaining a sample data set of a visual 3D target detection algorithm to be evaluated includes: obtaining a plurality of key frame sample images with a frequency of 2 Hz in the sample data set of the visual 3D target detection algorithm; obtaining annotation information of the original sample image by expanding the annotation information of the key frame sample image; and using the original sample image and the annotation information of the original sample image as the sample data set.

[0007] Optionally, the annotation information of the original sample image is obtained by expanding the annotation information of the key frame sample image, including: interpolating the annotation information of the key frame sample image to obtain an interpolated annotation dataset of the original sample image; constructing a time series annotation dataset of the original sample image based on the original sample point cloud and the key frame sample point cloud in the sample dataset and a preset detector; and generating the annotation information of the original sample image based on the interpolated annotation dataset and the time series annotation dataset.

[0008] Optionally, the interpolation processing of the annotation information of the key frame sample image to obtain the interpolated annotation data set of the original sample image includes: when the target object in the key frame sample image appears in two adjacent key frame sample images at the same time, generating the interpolated annotation data set according to the annotation information of the two adjacent key frame sample images.

[0009] Optionally, constructing the time series annotation dataset of the original sample image based on the original sample point cloud and key frame sample point cloud in the sample data set and a preset detector includes: training the preset detector based on the key frame sample point cloud; detecting the original sample point cloud using the trained preset detector to obtain the time series annotation dataset of the original sample point cloud; and sampling from the time series annotation dataset of the original sample point cloud to obtain the time series annotation dataset of the original sample image.

[0010] Optionally, generating the annotation information of the original sample image based on the interpolated annotation dataset and the time series annotation dataset includes: calculating the intersection-and-union ratio of the annotation information in the interpolated annotation dataset and the annotation information in the time series annotation dataset; if the intersection-and-union ratio is less than an intersection-and-union ratio threshold, merging the annotation information in the time series annotation dataset corresponding to the intersection-and-union ratio into the interpolated annotation dataset to obtain the annotation information of the original sample image.

[0011] Optionally, the method is applied to a test platform, and the method also includes: obtaining a running time distribution graph of the visual 3D target detection algorithm under preset hardware resources of the test platform; sampling the running time of the visual 3D target detection algorithm from the running time distribution graph; calculating the detection result output time of the visual 3D target detection algorithm for the target original sample image under simulated hardware resources based on the running time and the input time of the target original sample image in the sample data set; obtaining adjacent original sample images adjacent to the detection result output time in the sample data set; and calculating the average precision mean of the visual 3D target detection algorithm under the simulated hardware resources based on the detection result output time and the adjacent original sample images.

[0012] Optionally, obtaining the running time distribution graph of the visual 3D target detection algorithm under the preset hardware resources of the test platform includes: obtaining the running time distribution graph when running the target perception algorithm under the preset hardware resources of the test platform.

[0013] Optionally, the frequency of the key frame sample point cloud is 2 Hz, the frequency of the original sample point cloud is 20 Hz, the frequency of the original sample image is 12 Hz, and the frequency of the key frame sample image is 2 Hz.

[0014] An embodiment of the present invention also discloses an online evaluation system for a visual 3D target detection algorithm, the system comprising: a sample acquisition module, used to acquire a sample data set of a visual 3D target detection algorithm to be evaluated, the sample data set comprising a plurality of original sample images of preset frequencies and carrying annotation information, the preset frequency being greater than 2 Hz; an input-output module, used to input a first original sample image in the sample data set into the visual 3D target detection algorithm and output a detection result; an index evaluation module, used to perform index evaluation on the detection result and the annotation information of a second original sample image in the sample data set; wherein the second original sample image is the original sample image corresponding to the moment of outputting the detection result.

[0015] Optionally, the sample acquisition module includes: a key frame sample image acquisition module, used to acquire multiple key frame sample images with a frequency of 2 Hz in the sample data set of the visual 3D target detection algorithm; an annotation information expansion module, used to expand the annotation information of the original sample image according to the annotation information of the key frame sample image; and a sample data set determination module, used to use the original sample image and the annotation information of the original sample image as the sample data set.

[0016] Optionally, the annotation information expansion module includes: an interpolation module, used to interpolate the annotation information of the key frame sample image to obtain an interpolated annotation data set of the original sample image; a construction module, used to construct a time series annotation data set of the original sample image based on the original sample point cloud and the key frame sample point cloud in the sample data set and a preset detector; and a generation module, used to generate the annotation information of the original sample image based on the interpolated annotation data set and the time series annotation data set.

[0017] Optionally, the interpolation module is used to generate the interpolation annotation data set according to annotation information of two adjacent key frame sample images when the target object in the key frame sample image appears in two adjacent key frame sample images at the same time.

[0018] Optionally, the construction module includes: a training module, used to train the preset detector according to the key frame sample point cloud; a detection module, used to use the trained preset detector to detect the original sample point cloud to obtain a time-series annotation data set of the original sample point cloud; a sampling module, used to sample from the time-series annotation data set of the original sample point cloud to obtain a time-series annotation data set of the original sample image.

[0019] Optionally, the generation module includes: a calculation module, used to calculate the intersection-and-union ratio of the annotation information in the interpolated annotation data set and the annotation information in the time series annotation data set; and a merging module, used to merge the annotation information corresponding to the intersection-and-union ratio in the time series annotation data set into the interpolated annotation data set if the intersection-and-union ratio is less than an intersection-and-union ratio threshold, so as to obtain the annotation information of the original sample image.

[0020] Optionally, the system is applied to a test platform, and the system also includes: a distribution map acquisition module, which is used to obtain a running time distribution map of the visual 3D target detection algorithm under the preset hardware resources of the test platform; a running time sampling module, which is used to sample the running time of the visual 3D target detection algorithm from the running time distribution map; an output time calculation module, which is used to calculate the detection result output time of the visual 3D target detection algorithm for the target original sample image under the simulation hardware resources according to the running time and the input time of the target original sample image in the sample data set; an adjacent image acquisition module, which is used to obtain adjacent original sample images adjacent to the detection result output time in the sample data set; an average precision mean calculation module, which is used to calculate the average precision mean of the visual 3D target detection algorithm under the simulation hardware resources according to the detection result output time and the adjacent original sample images.

[0021] Optionally, the distribution graph acquisition module is used to obtain the runtime distribution graph when running the target perception algorithm under preset hardware resources of the test platform.

[0022] Optionally, the frequency of the key frame sample point cloud is 2 Hz, the frequency of the original sample point cloud is 20 Hz, the frequency of the original sample image is 12 Hz, and the frequency of the key frame sample image is 2 Hz.

[0023] An embodiment of the present invention also discloses an electronic device, comprising: one or more processors; and one or more machine-readable media having instructions stored thereon, which, when executed by the one or more processors, enables the electronic device to perform the online evaluation method of the visual 3D target detection algorithm as described above.

[0024] An embodiment of the present invention further discloses a computer-readable storage medium, which stores a computer program that enables a processor to execute the above-mentioned online evaluation method for a visual 3D object detection algorithm.

[0025] The embodiments of the present invention include the following advantages:

[0026] The online evaluation scheme for the visual 3D target detection algorithm provided by the embodiment of the present invention can first obtain a sample data set of the visual 3D target detection algorithm to be evaluated, and the sample data set can include multiple original sample images of preset frequencies, and the preset frequency is greater than 2 Hz. Then, the first original sample image in the sample data set is input into the visual 3D target detection algorithm, and the detection result of the first original sample image is output. Then, the detection result of the first original sample image and the annotation information of the second original sample image in the sample data set are evaluated by indicators. Among them, the second original sample image is the original sample image corresponding to the moment when the detection result of the first original sample image is output. In other words, the detection result of the first original sample image and the annotation information of the second original sample image at the current moment corresponding to the output detection result are evaluated by indicators.

[0027] In the embodiment of the present invention, the frequency of the original sample images of the sample data set is higher than the frequency of the sample data in the data set nuScenes, that is, the sample frequency of the visual 3D target detection algorithm is improved. When the running time of the visual 3D target detection algorithm is stable, the sample frequency is improved, which is equivalent to reducing the displacement deviation between the position of the target object in the first original sample image and the position of the target object in the second original sample image, so that the indicator evaluation of the visual 3D target detection algorithm is more in line with the real-time requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic diagram of the relationship between the detection results of a visual 3D target detection algorithm and a frame image;

[0029] Figure 2 is a flowchart of the steps of an online evaluation method of a visual 3D object detection algorithm according to an embodiment of the present invention;

[0030] Figure 3 is a flow chart of steps for generating annotation information of an original sample image according to an embodiment of the present invention;

[0031] Figure 4 is a running time distribution diagram of an embodiment of the present invention;

[0032] Figure 5 It is a flowchart of a real-time performance evaluation solution of a visual 3D target detection algorithm according to an embodiment of the present invention;

[0033] Figure 6It is a structural block diagram of an online evaluation system for a visual 3D target detection algorithm according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0035] The visual 3D target detection algorithm in the embodiment of the present invention can detect the target object in the data set nuScenes, and specifically can detect and obtain the position information of the target object in the image. The position information can be a 3D box. Since the target object may be moving in real time in the data set nuScenes, and it takes a certain amount of running time for the visual 3D target detection algorithm to detect the 3D box of the target object from a frame of image. Therefore, after the visual 3D target detection algorithm detects the target object in a frame of image and obtains the 3D box, the target object has moved to other locations, that is, the target object has appeared in another frame of image in the data set nuScenes. Refer to Figure 1 , which shows a schematic diagram of the relationship between the detection results of a visual 3D target detection algorithm and a frame image. Figure 1 In the figure, frame image A (Frame A), frame image B (Frame B) and frame image C (Frame C) are three consecutive frames of images of the target object (car) in the actual scene. Frame image A, frame image B and frame image C are input items of the visual 3D target detection algorithm and are input into the visual 3D target detection algorithm in sequence (Input time). When frame image A is input into the visual 3D target detection algorithm (Output time), the visual 3D target detection algorithm processes frame image A (Processing A). If the detection speed of the visual 3D target detection algorithm is not fast enough, when the target object moves to the position in frame image B, or even the position in frame image C, the visual 3D target detection algorithm will output the position of the target object in frame image A (Prediction of A, the 3D box in the figure is the predicted position of the target object). At this time, the target object has already appeared in the position in frame image C. If the position output by the visual 3D target detection algorithm is compared with the actual position of the target object in the frame image C, it is obvious that the output of the visual 3D target detection algorithm is the predicted position of the target object in the frame image A, that is, the predicted position of the target object before the frame image C (Previous prediction) is far behind the actual position of the target object in the frame image C. Therefore, the use of the nuScenes dataset to evaluate the visual 3D target detection algorithm cannot meet the real-time requirements.

[0036] Reference Figure 2, shows a flowchart of the steps of an online evaluation method for a visual 3D target detection algorithm according to an embodiment of the present invention. The online evaluation method for a visual 3D target detection algorithm may specifically include the following steps:

[0037] Step 201 : obtaining a sample data set of a visual 3D object detection algorithm to be evaluated.

[0038] In an embodiment of the present invention, the acquired sample data set includes multiple original sample images with preset frequencies and carrying annotation information. The preset frequency is greater than 2 Hz. The annotation information carried may be the position of the target object in the original sample image. The position may be a 3D box, etc.

[0039] In an embodiment of the present invention, multiple original sample images in a sample data set may form a continuous video stream. A target object may appear in one or more original sample images in a video stream. Moreover, the position of the target object in multiple original sample images may also vary. The embodiment of the present invention does not impose specific restrictions on the number, format, size, capacity, etc. of the original sample images in the sample data set. The embodiment of the present invention does not impose specific restrictions on the type, number, size, and position of the target object in the original sample image, etc.

[0040] Step 202: Input the first original sample image in the sample data set into the visual 3D target detection algorithm, and output the detection result.

[0041] In an embodiment of the present invention, the first original sample image may be any original sample image in the sample data set. Generally, the first original sample image may contain a target object. The output detection result is the predicted position of the target object in the first original sample image.

[0042] Step 203: perform an index evaluation on the detection result and the annotation information of the second original sample image in the sample data set.

[0043] In an embodiment of the present invention, the second original sample image is the original sample image corresponding to the moment of outputting the detection result. That is, the second original sample image is the original sample image at the current moment. For example, the first original sample image is image a, and when the visual 3D object detection algorithm outputs the detection result of image a, the original sample image corresponding to the sample data set is image c, and the detection result of image a and the annotation information of image c are evaluated.

[0044] The online evaluation scheme for the visual 3D target detection algorithm provided by the embodiment of the present invention can first obtain a sample data set of the visual 3D target detection algorithm to be evaluated, and the sample data set can include multiple original sample images of preset frequencies, and the preset frequency is greater than 2 Hz. Then, the first original sample image in the sample data set is input into the visual 3D target detection algorithm, and the detection result of the first original sample image is output. Then, the detection result of the first original sample image and the annotation information of the second original sample image in the sample data set are evaluated by indicators. Among them, the second original sample image is the original sample image corresponding to the moment when the detection result of the first original sample image is output. In other words, the detection result of the first original sample image and the annotation information of the second original sample image at the current moment corresponding to the output detection result are evaluated by indicators.

[0045] In the embodiment of the present invention, the frequency of the original sample images of the sample data set is higher than the frequency of the sample data in the data set nuScenes, that is, the sample frequency of the visual 3D target detection algorithm is improved. When the running time of the visual 3D target detection algorithm is stable, the sample frequency is improved, which is equivalent to reducing the displacement deviation between the position of the target object in the first original sample image and the position of the target object in the second original sample image, so that the indicator evaluation of the visual 3D target detection algorithm is more in line with the real-time requirements.

[0046] In an exemplary embodiment of the present invention, an implementation method of obtaining a sample data set of a visual 3D target detection algorithm to be evaluated is to obtain multiple key frame sample images with a frequency of 2 Hz in the sample data set of the visual 3D target detection algorithm; expand the annotation information of the original sample image according to the annotation information of the key frame sample image; and use the original sample image and the annotation information of the original sample image as the sample data set. In practical applications, the sample data set may include key frame sample images and original sample images. The frequency of the key frame sample image may be 2 Hz, and the frequency of the original sample image may be 12 Hz. Moreover, the key frame sample image also carries annotation information. However, the original sample image does not carry annotation information. Therefore, the embodiment of the present invention expands the annotation information of the original sample image according to the annotation information of the key frame sample image.

[0047] In an exemplary embodiment of the present invention, an implementation method of obtaining the annotation information of the original sample image based on the annotation information of the key frame sample image is to perform interpolation processing on the annotation information of the key frame sample image to obtain an interpolated annotation dataset of the original sample image. A time series annotation dataset of the original sample image is constructed based on the original sample point cloud and the key frame sample point cloud in the sample dataset and a preset detector. The annotation information of the original sample image is generated based on the interpolated annotation dataset and the time series annotation dataset.

[0048] In practical applications, one implementation method of interpolating the annotation information of the key frame sample images to obtain the interpolated annotation dataset of the original sample images is to generate the interpolated annotation dataset based on the annotation information of the two adjacent key frame sample images when the target object in the key frame sample images appears in two adjacent key frame sample images at the same time. e The key frame image corresponding to the time t s In the key frame image corresponding to the moment. Moreover, t e The key frame image corresponding to time t s The key frame images corresponding to the time are two adjacent key frame images. The annotation information of the two adjacent key frame sample images can be the pose information (3D position and 3D rotation angle) of the target object. Then, the pose information of the target object at time t is generated based on the pose information of the two adjacent key frame sample images. e <t<t s .

[0049]

[0050]

[0051] Among them, Tr(t) represents the 3D position of the target object at time t, Tr(t s ) indicates t s The 3D position of the target object at the moment, Tr(t e ) indicates t e The 3D position of the target object at time t. R(t) represents the 3D rotation angle of the target object at time t. s ) indicates t s The 3D rotation angle of the target object at the moment, R(t e ) indicates t e The 3D rotation angle of the target object at time t. Fs is the spherical linear interpolation equation. Tr(t) and R(t) together serve as the annotation information of the target object at time t.

[0052] Through the above interpolation method, an interpolation annotation dataset of the original sample image of 12 Hz can be obtained based on the annotation information of the key frame sample image of 2 Hz.

[0053] Theoretically, the above interpolation processing can expand the annotation information of the 2 Hz key frame sample image to the annotation information of the original sample image of 12 Hz. However, when the target object does not exist in the two key frame sample images of 2 Hz at the same time, the target object cannot obtain the annotation information of the intermediate moment corresponding to the two key frame sample images through interpolation processing. In order to solve this problem, the embodiment of the present invention constructs a time series annotation dataset.

[0054] In practical applications, one implementation method of constructing a time-series annotation dataset of the original sample image based on the original sample point cloud and the key frame sample point cloud in the sample data set and the preset detector is to train the preset detector based on the key frame sample point cloud. The original sample point cloud is detected using the trained preset detector to obtain a time-series annotation dataset of the original sample point cloud. The time-series annotation dataset of the original sample image is sampled from the time-series annotation dataset of the original sample point cloud. For example, the preset detector can use CenterPoint (a point cloud-based 3D detector). The sample data set can also include a key frame sample point cloud with a frequency of 2 Hz and an original sample point cloud with a frequency of 20 Hz. There are 10 frames of original sample point clouds between every two key frame sample images.

[0055] Reference Figure 3 , shows a flowchart of the steps of generating annotation information of an original sample image according to an embodiment of the present invention.

[0056] Step 301, training CenterPoint on the key frame sample point cloud.

[0057] Step 302: Detect the target object using the trained CenterPoint on the original sample point cloud to obtain a time-series annotation dataset of the original sample point cloud.

[0058] The time series annotation data set of the original sample point cloud is a 20 Hz time series annotation data set. The time series annotation data set of the original sample point cloud includes multiple moments and a 3D frame corresponding to each moment. Among them, the moment and the corresponding 3D frame can be understood as time series annotation data.

[0059] Step 303: Sample the 20 Hz time series annotation data set to obtain a 12 Hz time series annotation data set of the original sample image.

[0060] The time distribution of a 20 Hz frequency within one second is [0, 1 / 20, 2 / 20, 3 / 20, ...., 1], and the time distribution of a 12 Hz frequency within one second is [0, 1 / 12, 2 / 12, 3 / 12, ...., 1]. In order to sample a 12 Hz time series annotation dataset from a 20 Hz time series annotation dataset, the method of taking the time series annotation data at the closest moment can be used. For example, for the 1 / 12 moment in the 12 Hz time distribution, the closest time point to it in the 20 Hz time distribution is 2 / 20, so the time series annotation data corresponding to the 2 / 20 moment in the 20 Hz time distribution is sampled as the time series annotation data corresponding to the 1 / 12 moment in the 12 Hz time distribution.

[0061] Step 304 , redundant data in the time series annotation dataset and the interpolation mark dataset are removed according to the intersection-and-joint ratio, and finally the annotation information of the original sample image of 12 Hz is obtained by merging.

[0062] In practical applications, one implementation method of generating the annotation information of the original sample image based on the interpolation annotation dataset and the time series annotation dataset is to calculate the intersection over union of the annotation information in the interpolation annotation dataset and the annotation information in the time series annotation dataset; if the intersection over union is less than the intersection over union threshold, the annotation information corresponding to the intersection over union in the time series annotation dataset is merged into the interpolation annotation dataset to obtain the annotation information of the original sample image. For example, first select a 3D box in the 12 Hz time series annotation dataset, and calculate the intersection over union (IoU) of the 3D box and each 3D box in the 12 Hz interpolation annotation dataset. If IoU is greater than or equal to the intersection over union threshold (such as 0.7), it is considered that the 3D box for which the intersection over union is calculated appears in both the time series annotation dataset and the interpolation annotation dataset, and there is no need to merge the 3D box in the time series annotation dataset into the interpolation annotation dataset. If IoU is less than 0.7, it is considered that the 3D box in the time series annotation dataset does not appear in the interpolation annotation dataset, and the 3D box is merged into the interpolation annotation dataset. Equivalent to

[0063] In an exemplary embodiment of the present invention, the online evaluation scheme of the visual 3D target detection algorithm can be applied to a test platform. Specific hardware resources can be set for the test platform. For example, a graphics card (GPU) of a specified model is set for the test platform. The visual 3D target detection algorithm is run in a specific hardware resource environment to obtain a running time distribution graph of the visual 3D target detection algorithm. The running time distribution graph is as follows: Figure 4As shown. Wherein, the horizontal axis represents the running time of the visual 3D target detection algorithm, and the vertical axis represents the probability of the running time of the visual 3D target detection algorithm. In the actual performance evaluation process, it is not necessary to evaluate the visual 3D target detection algorithm in a variety of real environments, and the performance evaluation can be performed in a simulation environment, that is, under simulated hardware resources. Specifically, the running time of the visual 3D target detection algorithm is obtained by random sampling from the running time distribution diagram, and then the detection result output time of the visual 3D target detection algorithm for the target original sample image under the simulated hardware resources is calculated according to the running time and the input time of the target original sample image in the sample data set. Obtain the adjacent original sample images adjacent to the detection result output time in the sample data set; calculate the mean average precision (meanAverage Precision, referred to as mAP) of the visual 3D target detection algorithm under the simulated hardware resources according to the detection result output time and the adjacent original sample images. Wherein, assuming that the running time obtained by random sampling is m, the input time corresponding to the target original sample image is t, and the calculated detection result output time of the visual 3D target detection algorithm for the target original sample image under the simulated hardware resources is m+t. Then, the original sample image at the time closest to m+t (ie, the adjacent original sample image) is obtained in the 12 Hz sample data set.

[0064] In an exemplary embodiment of the present invention, the test platform can also be shared with other perception algorithms to evaluate the performance of the visual 3D target detection algorithm in real time under the shared test platform. That is to say, when obtaining the running time distribution graph of the visual 3D target detection algorithm under the preset hardware resources of the test platform, the target perception algorithm (such as classification algorithm, 2D detection algorithm, object segmentation algorithm, etc.) can be run simultaneously under the preset hardware resources. The running speed of the visual 3D target detection algorithm will slow down. At this time, the running time distribution graph of the visual 3D target detection algorithm is obtained, and then random sampling is performed on the running time distribution graph to perform real-time performance evaluation under simulated hardware resources.

[0065] Based on the above description of an embodiment of an online evaluation method for a visual 3D object detection algorithm, a real-time performance evaluation scheme for a visual 3D detection algorithm is introduced below. Figure 5 , shows a flow chart of a real-time performance evaluation scheme for a visual 3D target detection algorithm according to an embodiment of the present invention.

[0066] Step 501 : Input a surround view image set with a frequency of 12 Hz into a visual 3D target detection algorithm.

[0067] Step 502, setting the hardware resources of the test platform where the visual 3D object detection algorithm is located.

[0068] Step 503: Obtain the detection result of the visual 3D target detection algorithm for the surround view image at a certain moment.

[0069] Step 504: perform an index evaluation on the detection result and the annotation information of the surround view image corresponding to the current moment.

[0070] Step 505, determining whether there is a next frame of surround view image.

[0071] If there is no next frame of surround view image, the solution ends; if there is a next frame of surround view image, step 506 is executed.

[0072] Step 506: Input the next frame of surround view image to the visual 3D target detection algorithm. Step 503 is executed repeatedly.

[0073] The embodiment of the present invention proposes a solution for real-time evaluation of a visual 3D object detection algorithm. Compared with offline performance evaluation, the indicators obtained by real-time evaluation are more suitable for the performance of the visual 3D object detection algorithm when applied in the real world.

[0074] The embodiment of the present invention expands the annotation information of the 2 Hz key frame sample images in the data set nuScenes to the annotation information of the original sample images of 12 Hz, provides annotation information for sample images with higher frequency, and forms a sample data set that is more suitable for real-time evaluation.

[0075] The embodiment of the present invention performs a visual 3D object detection algorithm performance evaluation in a fixed hardware resource environment, and then performs random sampling according to a run time distribution graph, so as to perform a visual 3D object detection algorithm performance evaluation in a simulated hardware resource environment.

[0076] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0077] Reference Figure 6 , shows a structural block diagram of an online evaluation system for a visual 3D target detection algorithm according to an embodiment of the present invention. The online evaluation system for a visual 3D target detection algorithm may specifically include the following modules.

[0078] A sample acquisition module 61 is used to acquire a sample data set of a visual 3D object detection algorithm to be evaluated, wherein the sample data set includes a plurality of original sample images with preset frequencies and carrying annotation information, wherein the preset frequency is greater than 2 Hz;

[0079] An input-output module 62, configured to input the first original sample image in the sample data set into the visual 3D object detection algorithm and output a detection result;

[0080] An index evaluation module 63, used to perform index evaluation on the detection result and the annotation information of the second original sample image in the sample data set;

[0081] The second original sample image is the original sample image corresponding to the moment when the detection result is output.

[0082] In an exemplary embodiment of the present invention, the sample acquisition module 61 includes:

[0083] A key frame sample image acquisition module, used to acquire a plurality of key frame sample images with a frequency of 2 Hz in the sample data set of the visual 3D target detection algorithm;

[0084] A labeling information expansion module, used to expand the labeling information of the key frame sample image to obtain the labeling information of the original sample image;

[0085] The sample data set determination module is used to use the original sample image and the annotation information of the original sample image as the sample data set.

[0086] In an exemplary embodiment of the present invention, the annotation information expansion module includes:

[0087] An interpolation module, used for performing interpolation processing on the annotation information of the key frame sample image to obtain an interpolation annotation data set of the original sample image;

[0088] A construction module, used to construct a time-series annotation dataset of the original sample image according to the original sample point cloud and the key frame sample point cloud in the sample data set and a preset detector;

[0089] A generating module is used to generate the annotation information of the original sample image according to the interpolation annotation data set and the time series annotation data set.

[0090] In an exemplary embodiment of the present invention, the interpolation module is used to generate the interpolation annotation data set according to the annotation information of the two adjacent key frame sample images when the target object in the key frame sample image appears in two adjacent key frame sample images at the same time.

[0091] In an exemplary embodiment of the present invention, the building block comprises:

[0092] A training module, used for training the preset detector according to the key frame sample point cloud;

[0093] A detection module, used to detect the original sample point cloud using the trained preset detector to obtain a time-series annotation data set of the original sample point cloud;

[0094] The sampling module is used to sample the time-series annotation data set of the original sample point cloud to obtain the time-series annotation data set of the original sample image.

[0095] In an exemplary embodiment of the present invention, the generating module comprises:

[0096] A calculation module, used to calculate the intersection-over-union ratio of the annotation information in the interpolation annotation data set and the annotation information in the time series annotation data set;

[0097] A merging module is used to merge the annotation information corresponding to the IoU ratio in the time series annotation dataset into the interpolation annotation dataset if the IoU ratio is less than an IoU ratio threshold, so as to obtain the annotation information of the original sample image.

[0098] In an exemplary embodiment of the present invention, the system is applied to a test platform, and the system further comprises:

[0099] A distribution graph acquisition module, used to obtain a running time distribution graph of the visual 3D object detection algorithm under the preset hardware resources of the test platform;

[0100] A running time sampling module, used for sampling the running time of the visual 3D object detection algorithm from the running time distribution graph;

[0101] An output time calculation module, used to calculate the output time of the detection result of the visual 3D target detection algorithm for the target original sample image under the simulation hardware resources according to the running time and the input time of the target original sample image in the sample data set;

[0102] An adjacent image acquisition module, used to acquire adjacent original sample images in the sample data set that are adjacent to the output time of the detection result;

[0103] The average precision mean calculation module is used to calculate the average precision mean of the visual 3D target detection algorithm under the simulation hardware resources according to the detection result output time and the adjacent original sample images.

[0104] In an exemplary embodiment of the present invention, the distribution graph acquisition module is used to acquire the runtime distribution graph when the target perception algorithm is run under the preset hardware resources of the test platform.

[0105] In an exemplary embodiment of the present invention, the frequency of the key frame sample point cloud is 2 Hz, the frequency of the original sample point cloud is 20 Hz, the frequency of the original sample image is 12 Hz, and the frequency of the key frame sample image is 2 Hz.

[0106] As for the system embodiment and the device embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0107] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0108] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0109] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0110] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0111] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0112] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0113] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0114] The above is a detailed introduction to an online evaluation method for a visual 3D target detection algorithm and an online evaluation system for a visual 3D target detection algorithm provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technicians in this field, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for online evaluation of a visual 3D object detection algorithm, characterized in that: The method comprises: Obtain a sample data set of a visual 3D target detection algorithm to be evaluated, wherein the sample data set includes a plurality of original sample images with preset frequencies and carrying annotation information, wherein the preset frequency is greater than 2 Hz, and the annotation information is the position of the target object in the original sample image; Inputting the first original sample image in the sample data set into the visual 3D target detection algorithm, and outputting a detection result; wherein the detection result is the predicted position of the target object in the first original sample image; Performing an index evaluation on the detection result and the annotation information of the second original sample image in the sample data set to evaluate the performance of the visual 3D object detection algorithm; The second original sample image is the original sample image corresponding to the moment when the detection result is output.

2. The method according to claim 1, characterized in that The step of obtaining a sample data set of a visual 3D object detection algorithm to be evaluated includes: Acquire a plurality of key frame sample images with a frequency of 2 Hz in the visual 3D target detection algorithm sample data set; Expanding the annotation information of the key frame sample image to obtain the annotation information of the original sample image; The original sample image and the annotation information of the original sample image are used as the sample data set.

3. The method according to claim 2, characterized in that The step of expanding the annotation information of the original sample image according to the annotation information of the key frame sample image includes: Performing interpolation processing on the annotation information of the key frame sample image to obtain an interpolation annotation data set of the original sample image; Constructing a time-series annotation dataset of the original sample image according to the original sample point cloud and the key frame sample point cloud in the sample data set and a preset detector; The annotation information of the original sample image is generated according to the interpolation annotation data set and the temporal annotation data set.

4. The method according to claim 3, characterized in that The interpolation processing of the annotation information of the key frame sample image to obtain the interpolation annotation data set of the original sample image includes: When the target object in the key frame sample image appears in two adjacent key frame sample images at the same time, the interpolation annotation data set is generated according to the annotation information of the two adjacent key frame sample images.

5. The method according to claim 3, characterized in that: The step of constructing a temporal annotation dataset of the original sample image according to the original sample point cloud and the key frame sample point cloud in the sample data set and a preset detector includes: Training the preset detector according to the key frame sample point cloud; Detecting the original sample point cloud using the trained preset detector to obtain a time-series annotation data set of the original sample point cloud; The time-series annotation dataset of the original sample image is obtained by sampling from the time-series annotation dataset of the original sample point cloud.

6. The method according to claim 3, characterized in that The step of generating the annotation information of the original sample image according to the interpolation annotation dataset and the time series annotation dataset includes: Calculating an intersection-over-union ratio of the annotation information in the interpolation annotation dataset and the annotation information in the time series annotation dataset; If the IoU ratio is less than an IoU ratio threshold, the annotation information corresponding to the IoU ratio in the time series annotation dataset is merged into the interpolation annotation dataset to obtain the annotation information of the original sample image.

7. The method according to claim 1, characterized in that The method is applied to a test platform, and the method further comprises: Obtaining a running time distribution diagram of the visual 3D object detection algorithm under the preset hardware resources of the test platform; Sampling the runtime of the visual 3D object detection algorithm from the runtime distribution graph; Calculate the output time of the detection result of the visual 3D object detection algorithm for the target original sample image under the simulation hardware resources according to the running time and the input time of the target original sample image in the sample data set; Acquire adjacent original sample images in the sample data set that are adjacent to the output time of the detection result; The average precision mean of the visual 3D object detection algorithm under the simulation hardware resources is calculated according to the detection result output time and the adjacent original sample images.

8. The method according to claim 7, characterized in that The obtaining of a running time distribution diagram of the visual 3D object detection algorithm under the preset hardware resources of the test platform includes: When the target perception algorithm is run under the preset hardware resources of the test platform, the running time distribution graph is obtained.

9. The method according to claim 3, characterized in that: The frequency of the key frame sample point cloud is 2 Hz, the frequency of the original sample point cloud is 20 Hz, the frequency of the original sample image is 12 Hz, and the frequency of the key frame sample image is 2 Hz.

10. A visual 3D object detection algorithm online evaluation system, characterized in that: The system comprises: A sample acquisition module is used to acquire a sample data set of a visual 3D target detection algorithm to be evaluated, wherein the sample data set includes a plurality of original sample images with preset frequencies and carrying annotation information, wherein the preset frequency is greater than 2 Hz, and the annotation information is the position of the target object in the original sample image; An input-output module, used to input the first original sample image in the sample data set into the visual 3D target detection algorithm, and output a detection result; wherein the detection result is the predicted position of the target object in the first original sample image An index evaluation module, used to perform an index evaluation on the detection result and the annotation information of the second original sample image in the sample data set to evaluate the performance of the visual 3D object detection algorithm; The second original sample image is the original sample image corresponding to the moment when the detection result is output.

11. An electronic device, characterized in that: include: one or more processors; and One or more machine-readable media having instructions stored thereon, when executed by the one or more processors, enable the electronic device to perform the online evaluation method of the visual 3D target detection algorithm as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer program stored therein enables the processor to execute the online evaluation method for a visual 3D target detection algorithm as described in any one of claims 1 to 9.

Images