Test method and system for algorithm of unmanned mine car and program product

Abstract

The invention provides a test method and system for an algorithm of an unmanned mine car and a program product, and relates to the field of algorithm test. The test method of the perceptual visual identification algorithm for the unmanned mine car comprises the following steps: inputting a pre-stored sensor data set into a perceptual visual identification algorithm to be tested to output an algorithm calculation result; data processing is performed on the algorithm calculation result to obtain object identification information, statistical processing is performed on the object identification information to obtain a final identification result, and the final identification result comprises output results of the multiple test items; and comparing the final recognition result with a pre-stored reference recognition result to obtain a test result of the perceptual visual recognition algorithm. According to the method and the system, on-site manual testing can be avoided.

Description

Technical Field

[0001] The present invention relates to a testing method, a testing system and a computer program product for a perception visual recognition algorithm of an unmanned mining vehicle. Background Art

[0002] In the development and application of algorithm software, testing is an important part of ensuring the quality and stability of the algorithm. For the perception visual recognition algorithm applied to unmanned mining vehicles, the traditional method is to conduct on-site manual testing. However, although on-site manual testing is accurate in some aspects, this type of testing requires a lot of manpower, time and energy, and may still not be able to avoid errors and omissions. In addition, during the long testing process, personnel may be fatigued, resulting in inaccurate judgments or missed tests. This not only wastes time and resources, but also has a significant impact on the entire testing process.

[0003] On the other hand, since on-site manual testing requires a lot of manpower and time resources, batch testing is not possible. This means that multiple samples cannot be tested at the same time, which leads to a longer testing cycle and limits the production and time to market of the product. Therefore, it is necessary to introduce advanced automated testing methods to effectively solve the challenges faced by on-site testing. Summary of the invention

[0004] The purpose of the present invention is to propose a testing method, a testing system and a computer program product for a perception visual recognition algorithm for an unmanned mining vehicle, which can realize batch testing in multiple scenarios, and has low testing cost, high testing efficiency and good testing stability.

[0005] According to one aspect of the present invention, there is provided a testing method for a perceptual visual recognition algorithm for an unmanned mining vehicle, the method comprising: inputting a pre-stored sensor data set into the perceptual visual recognition algorithm to be tested to output an algorithm calculation result; performing data processing on the algorithm calculation result to obtain object recognition information, and performing statistical processing on the object recognition information to obtain a final recognition result, the final recognition result comprising output results of a plurality of test items; and comparing the final recognition result with a pre-stored reference recognition result to obtain a test result of the perceptual visual recognition algorithm.

[0006] In an embodiment, the sensor data set is acquired by at least one visual sensor used for the unmanned mining vehicle at multiple time frames, so that the sensor data set includes multiple time frames and sensor data corresponding to each time frame; and the perceptual visual recognition algorithm performs calculations on the sensor data of each time frame respectively to obtain the algorithm calculation results.

[0007] In an embodiment, performing data processing on the algorithm calculation results includes: performing format conversion on the algorithm calculation results, extracting the identified object from the converted algorithm calculation results, and obtaining relevant information of the identified object as the multiple test items, and storing the multiple test items in corresponding time frames to form the object recognition information.

[0008] In an embodiment, performing statistical processing on the object recognition information includes: for each test item, performing percentage statistics in all time frames, so as to use the acquired information with the largest percentage as the output result of the corresponding test item, and record it in the final recognition result.

[0009] In an embodiment, the reference identification result includes reference results of multiple test items; and comparing the output results of the multiple test items with the reference results of the multiple test items includes: matching and comparing the output results with the reference results for each test item respectively to obtain test items that have been successfully detected, test items that have failed to be detected, test items that have been missed, and test items that have been incorrectly identified and recording them in the test result, and calculating the success rate based on the successfully identified test items.

[0010] In an embodiment, the testing method further includes: determining a time frame of an output result including a detection failure in the object recognition information based on the reference recognition result, and recording and feeding back the determined time frame and designated information associated with the determined time frame.

[0011] In an embodiment, the sensor data set also includes scene information, and the testing method further includes: inputting multiple sensor data sets collected in different multiple scenes into the perceptual visual recognition algorithm to be tested to obtain the test results for different scenes.

[0012] In an embodiment, the reference recognition result includes scene information, and comparing the final recognition result with a pre-stored reference recognition result also includes: obtaining a reference recognition result with the same scene information based on the scene information in the input sensor data set, so as to compare the final recognition result with the reference recognition result based on the same scene.

[0013] In an embodiment, statistical processing is performed on the test results of the multiple scenarios to obtain at least one of a detection success rate, a detection omission rate, and an error recognition rate of the test items.

[0014] In an embodiment, the multiple scenes include at least one of a scene based on time changes, a scene based on light changes, a scene based on distance changes, a scene based on object state changes, a scene based on object type changes, and a scene based on object quantity changes.

[0015] According to another aspect of the present invention, a testing system for a perception visual recognition algorithm of an unmanned mining vehicle is provided, the testing system executing a testing method according to an embodiment of the present invention, the testing system comprising: a publishing module, the publishing module being configured to input a sensor data set into the perception visual recognition algorithm to be tested, the sensor data set being acquired by at least one visual sensor for the unmanned mining vehicle at multiple time frames and being pre-stored in the publishing module; an algorithm execution module, the algorithm execution module being configured to enable the perception visual recognition algorithm to perform calculations based on the input sensor data set to output algorithm calculation results; and a subscription module, the subscription module being configured to perform data processing on the algorithm calculation results to obtain object recognition information, and to perform statistical processing on the object recognition information to obtain a final recognition result, the final recognition result comprising output results of multiple test items.

[0016] In an embodiment, the subscription module includes a data processing unit, which is configured to perform format conversion on the algorithm calculation results, extract the identified objects from the converted algorithm calculation results, and obtain relevant information of the identified objects as the multiple test items, and store the multiple test items in corresponding time frames to form the object identification information.

[0017] In an embodiment, the subscription module further includes a data statistics unit, which is configured to perform percentage statistics in all time frames for each test item, so as to use the acquired information with the largest percentage as the output result of the corresponding test item.

[0018] In an embodiment, the publishing module is also configured to send pre-stored reference identification results to the subscription module, and the reference identification results include reference results of multiple test items; and the subscription module also includes a comparison unit, which is configured to match and compare the output results with the reference results for each test item respectively, so as to obtain test items that have been successfully detected, test items that have failed to be detected, test items that have been missed, and test items that have been incorrectly identified and record them in the test results, and calculate the success rate based on the successfully identified test items.

[0019] In an embodiment, the subscription module also includes a feedback unit, which is configured to determine a time frame of the output results including detection failure in the object recognition information based on the reference recognition result, and record and feedback the determined time frame and specified information associated with the determined time frame.

[0020] According to yet another aspect of the present invention, a computer program product is provided. The computer program product stores computer instructions, and the computer instructions are used to enable a computer to execute a method according to an embodiment of the present invention.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present invention.

[0023] Figure 1 A schematic diagram of a testing system for a perception visual recognition algorithm for an unmanned mining vehicle according to an embodiment of the present invention is shown.

[0024] Figure 2 A schematic diagram of a subscription module according to an embodiment of the present invention is shown.

[0025] Figure 3 An example of object recognition information according to an embodiment of the present invention is shown.

[0026] Figure 4 An example of a final recognition result and a reference recognition result according to an embodiment of the present invention is shown.

[0027] Figure 5 A flow chart of a method for testing a perception visual recognition algorithm for an unmanned mining vehicle according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0028] In order to more clearly explain the purpose, technical solutions and advantages of the present invention, the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. It should be understood that the following description of the embodiments is intended to explain and illustrate the overall concept of the present invention, and should not be understood as limiting the present invention. In the specification and the drawings, the same or similar reference numerals refer to the same or similar parts or components. For the sake of clarity, the drawings are not necessarily drawn to scale, and some well-known parts and structures may be omitted in the drawings.

[0029] Unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. The wording "one" or "an" does not exclude multiple. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right", "top" or "bottom" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly. When an element such as a layer, a film, a region or a substrate substrate is referred to as being "on" or "under" another element, the element may be "directly" located "on" or "under" another element, or there may be an intermediate element.

[0030] Figure 1 FIG. 1 is a schematic diagram of a testing system 1000 for a perception visual recognition algorithm for an unmanned mining vehicle according to an embodiment of the present invention. Figure 1 As shown, the test system 1000 may include a publishing module 10 , an algorithm execution module 20 , and a subscription module 30 .

[0031] The publishing module 10 may pre-store a sensor data set CDS, and may be configured to input the pre-stored sensor data set CDS into the algorithm execution module 20. In this article, the "sensor data set CDS" may be understood as a data set obtained by collecting data in a mining scene by at least one visual sensor for an unmanned mining vehicle. For example, a mining scene and location may be pre-specified, and then the unmanned mining vehicle may be driven to the mining location, and at least one visual sensor on the unmanned mining vehicle may be used to collect data for the mining scene, thereby obtaining a sensor data set CDS corresponding to the mining scene. According to an embodiment of the present invention, data may be collected in multiple scenes in advance to obtain multiple sensor data sets CDS. Then, all the collected sensor data sets CDS may be stored in the publishing module 10 for use in subsequent algorithm testing.

[0032] According to an embodiment of the present invention, the sensor data set CDS may include sensor data collected by at least one visual sensor of the driverless mining vehicle at multiple time frames. For example, in the process of collecting data, the at least one visual sensor of the driverless mining vehicle may continuously shoot multiple time frames to obtain sensor data corresponding to each time frame. Therefore, the sensor data set CDS may include multiple time frames and sensor data corresponding to each time frame. As an example, each time frame may be 1 millisecond (ms).

[0033] The algorithm execution module 20 may be configured to receive the perceptual visual recognition algorithm ALO to be tested. In this article, the "perceptual visual recognition algorithm ALO" may be understood as a control system designed to be suitable for an unmanned mine car, and capable of processing data collected by at least one visual sensor of the unmanned mine car, thereby obtaining an algorithm for the driving conditions of the surrounding roads of the mining area, thereby helping the unmanned mine car to avoid and bypass obstacles. According to an embodiment of the present invention, the algorithm execution module 20 may receive multiple perceptual visual recognition algorithms ALO, and optionally delete or replace the received perceptual visual recognition algorithms ALO. For example, after testing the perceptual visual recognition algorithm ALO, if the perceptual visual recognition algorithm ALO test fails and needs to be adjusted, the adjusted algorithm may be input into the algorithm execution module 20 after updating for a new test. For another example, after updating the perceptual visual recognition algorithm ALO for reasons such as matching the usage scenario or usage needs, it may also be directly input into the algorithm execution module 20 for testing.

[0034] As an example, when performing a test, the perceptual visual recognition algorithm ALO to be tested may be first uploaded to the algorithm execution module 20, and then the publishing module 10 may input the sensor data set CDS into the algorithm execution module 20. According to an embodiment of the present invention, the algorithm execution module 20 may be configured to enable the perceptual visual recognition algorithm ALO to perform calculations based on the input sensor data set CDS to output an algorithm calculation result ARE. As an example, as described above, the sensor data set CDS includes a plurality of sensor data corresponding to a plurality of time frames, respectively. The perceptual visual recognition algorithm ALO may perform calculations for the sensor data under each time frame, respectively, thereby obtaining a plurality of calculation results corresponding to a plurality of time frames, and thereby forming the algorithm calculation result ARE.

[0035] The subscription module 30 may be configured to perform data processing and statistical processing on the algorithm calculation result ARE to obtain the final recognition result FRE. The publishing module 10 may also be configured to pre-store a reference recognition result RRE and send the reference recognition result RRE to the subscription module 30, so that the test result of the perceptual visual recognition algorithm ALO is obtained by comparing the final recognition result FRE with the reference recognition result RRE.

[0036] In this way, the ALO perception visual recognition algorithm can be tested without going to the mine site, thus solving the technical problems of high cost and low efficiency of on-site testing. In addition, since there is no need for on-site testing, there is no need to consume a lot of manpower and time resources, so that batch testing of multiple samples can be carried out at the same time without extending the test cycle. Thus, the effectiveness and accuracy of the test are improved.

[0037] Figure 2 FIG. 2 shows a schematic diagram of a subscription module 30 according to an embodiment of the present invention. Figure 2 As shown, the subscription module 30 includes a data processing unit 310 , a data statistics unit 320 , a comparison unit 330 and a feedback unit 340 .

[0038] According to an embodiment of the present invention, the data processing unit 310 may perform data processing on the algorithm calculation result ARE to obtain the object identification information OED. The data processing unit 310 may be configured to perform format conversion on the algorithm calculation result ARE to extract the identified object from the converted algorithm calculation result ARE, obtain the related information of the identified object as multiple test items, store the multiple test items under the corresponding time frame to form the object identification information OED. As an example, after the perception visual recognition algorithm ALO performs calculations, the objects existing in the collected scene (or at least the objects that the unmanned mine car must pay attention to during driving) and the related information of the objects (for example, the type of the object, the distance from the unmanned mine car, the size, etc.) may be calculated and included in the algorithm calculation result ARE. However, since the perception visual recognition algorithm ALO is configured to be used in conjunction with the control system of the unmanned mine car, the format of the algorithm calculation result ARE output from the perception visual recognition algorithm ALO may not be suitable or convenient for the test system 1000 to perform subsequent processing. The data processing unit 310 may perform format conversion on the algorithm calculation result ARE to convert it into a data format suitable for subsequent processing. Then, the data processing unit 310 can determine and extract the one or more objects identified from the algorithm calculation results ARE after format conversion, and obtain information related to each object. In this way, through the format conversion process and extraction process of the data processing unit 310, the identified objects and their related information can be represented in a desired format or form, so that the subsequent statistical process can be facilitated.

[0039] Figure 3 An example of object identification information OED according to an embodiment of the present invention is shown. As an example, Figure 3 It is shown that three objects (i.e., object 1-object 3) are extracted and recognized from the algorithm calculation results based on three time frames (i.e., time frame 1-time frame 3), and three types of information (e.g., type, size, distance) of each object are obtained as test items (i.e., T1-T9). However, this is only an example, and embodiments of the present invention are not limited thereto.

[0040] like Figure 3As shown, the type information of object 1 can be stored as test item T1, the distance information of object 1 can be stored as test item T2, the size information of object 1 can be stored as test item T3, the type information of object 2 can be stored as test item T4, the distance information of object 2 can be stored as test item T5, the size information of object 2 can be stored as test item T6, the type information of object 3 can be stored as test item T7, the distance information of object 3 can be stored as test item T8, and the size information of object 3 can be stored as test item T9. As described above, the algorithm calculation result ARE includes the calculation results corresponding to each time frame, and therefore includes the object-related information in each time frame, that is, the information of each test item T1-T9 in each time frame. After the data processing unit 310 obtains this information from the algorithm calculation result ARE, the obtained information can be stored in the corresponding time frames to form object identification information OED. For example, Figure 3 As shown, for test item T1, it is identified as a human type in time frame 1-time frame 3; for test item T2, it is identified as distance L1 in time frame 1-time frame 2, but is identified as distance L2 in time frame 3; for test item T3, it is identified as size S1 in time frame 1-time frame 3. Similarly, in this way, the acquired information (e.g., size S3, S4, etc.) of each test item T4-T9 in time frame 1-time frame 3 can be clearly known, so as to facilitate the subsequent statistical processing process. However, it should be noted that the relevant information of the identified object may not be limited to type, size, and distance, and may include more other information according to the nature of the algorithm and actual needs.

[0041] The object recognition information OED may be statistically processed by the data statistics unit 320. The data statistics unit 320 may be configured to perform proportion statistics in all time frames for each test item, so as to take the acquired information with the largest proportion as the output result of the corresponding test item, and record it in the final recognition result FRE.

[0042] Figure 4 1 shows an example of a final recognition result FRE and a reference recognition result RRE according to an embodiment of the present invention. Figure 3-4 As shown, in the formation Figure 3After the object identification information OED is obtained, the data statistics unit 320 performs proportion statistics processing for each test item T1-T9 respectively to determine the acquisition information with the largest proportion for each test item. For example, for test item T1, since it is identified as a human type in time frames 1-3, the acquisition information with the largest proportion is the human type. For test item T2, since it is identified as distance L1 in two time frames (i.e., time frame 1 and time frame 2) and is only identified as distance L2 in one time frame (i.e., time frame 3), the acquisition information with the largest proportion is distance L1. For test item T3, since it is identified as size S1 in time frames 1-3, the acquisition information with the largest proportion is size S1. Similarly, the maximum proportion information of test item T4 is the stone type, the maximum proportion information of test item T5 is the distance L3, the maximum proportion information of test item T6 is the size S4, the maximum proportion information of test item T7 is the mine car type, the maximum proportion information of test item T8 is the distance L5, and the maximum proportion information of test item T8 is the size S5. Therefore, the determined maximum proportion information of each test item can be used as the output result of each test item, thereby forming the final recognition result FRE. Therefore, if Figure 4 As shown, the respective output results of the test items T1-T9 in the final recognition result FRE are: human type, L1, S1, stone type, L3, S4, mine cart type, L5, S5.

[0043] However, it should be noted that the above content is only an example, and only 3 time frames and 9 test items are shown for ease of understanding. In actual applications, there may be more time frames and test items. In addition, in the calculation of the percentage, various suitable methods can be used to determine the information with the maximum percentage. In addition to determining the percentage by counting the number of each type of information, the information with the maximum percentage can also be determined by setting a threshold. For example, the threshold can be set to 40%, and when the number of time frames with a type of acquired information accounts for more than 40% of the number of all time frames, the acquired information is considered to be the information with the maximum percentage. In other words, the calculation method of the maximum percentage information can be predefined in any suitable manner according to actual conditions, as long as the obtained result represents the test trend of the corresponding identification item.

[0044] like Figure 4As shown, the reference recognition result RRE may include reference results of multiple test items. As mentioned above, the reference recognition result RRE is also pre-stored in the publishing module 10. In this article, "reference recognition result RRE" can be understood as reference information that is simultaneously acquired and recorded when the sensor data set CDS is collected, so that when the perception visual recognition algorithm ALO performs calculations based on the sensor data set CDS, the corresponding reference information can be used for comparison to determine whether the test passes. For example, the reference recognition result RRE may include objects that actually exist on the road surface and related information thereof, or may include objects that the reference recognition result RRE should identify and related information thereof. As an example, Figure 4 The reference recognition result RRE shown in FIG. 1 includes reference results of 12 test items T1-T12 of 4 objects, and the reference recognition result RRE corresponds to the calculated Figure 4 In other words, the publishing module 10 can input the sensor data set CDS into the perceptual visual recognition algorithm ALO so that the subscription module 30 can calculate the final recognition result FRE (for example, Figure 4 The final recognition result FRE is obtained by converting the reference recognition result RRE (for example, Figure 4 The reference recognition result RRE) is sent to the subscription module 30, thereby comparing the reference recognition result RRE with the final recognition result FRE to output the test result TR.

[0045] The comparison between the reference recognition result RRE and the final recognition result FRE may be performed by the comparison unit 330. The comparison unit 330 may be configured to match and compare the output result with the reference result for each test item, so as to obtain the test items that have been successfully detected, the test items that have failed to be detected, the test items that have been missed in detection, and the test items that have been incorrectly identified, and record them in the test result TR. As an example, Figure 4 As shown, the recognized object in the final recognition result FRE can be matched with the reference object in the reference recognition result RRE according to the object features, so as to match the reference result and output result of each test item one by one. Figure 4 It can be seen that the reference recognition result RRE includes 4 objects, so through comparison, it can be obtained that object 4 and the corresponding test items T10-T12 (the reference recognition result RRE is stone, L7, S7) are missed, so the test items T10-T12 can be recorded in the test result TR as the test items missed by detection. In another embodiment, all existing objects may be recognized, but all relevant information may not be output. For example, object 4 may be recognized, but the test item T12 may not be output, and the test item T12 may be recorded as the test item missed by detection in the test result TR.

[0046] For test items T1-T9, the output results can be matched and compared with the reference results respectively. Figure 4 As shown, the output result of test item T2 is L1, but the reference result is L2, so it can be determined that test item T12 is not detected successfully, that is, it can be recorded in the test result TR as a test item that failed to be detected. On the other hand, for test item T1 and test items T3-T9, the output result and the reference result can match each other, so it can be determined that test item T1 and test items T3-T9 are detected successfully, that is, they can be recorded in the test result TR as test items that are successfully detected. In another embodiment, there may be an object or a test item included in the final recognition result FRE, but not included in the reference recognition result RRE. In this case, it can be determined that such a test item or the test item corresponding to such an object is misdetected, and can be recorded in the test result TR as a misidentified test item.

[0047] In this way, the test result TR can record the test items that have been successfully detected, the test items that have failed to be detected, the test items that have been missed, and the test items that have been incorrectly identified, so that the tester can clearly and quickly understand the error aspects and performance of the tested algorithm, and thus can improve the tested algorithm in a targeted manner. Further, the comparison unit 330 can also be configured to calculate the success rate based on the test result TR. For example, the success rate = the number of test items that have been successfully detected / the number of all test items included in the reference recognition result RRE × 100%.

[0048] The subscription module 30 also includes a feedback unit 340. The feedback unit 340 can be configured to determine the time frame of the output results including the detection failure in the object recognition information OED based on the reference recognition result RRE, and record and feedback the determined time frame and the specified information associated with the determined time frame. As an example, the feedback unit 340 can match and compare the information of each test item in each time frame with the information of each test item in the reference recognition result RRE, and when unmatched test item information is found, determine the time frame in which the test item information is located, and obtain the specified information associated with the time frame, such as the picture collected in the time frame, the specific time information corresponding to the time frame, etc. Then, the time frame and the specified information are recorded and fed back to the tester together. For example, Figure 3 The information of the test item T2 in the time frame 1 is L1, but the information of the test item T2 in the reference recognition result RRE is L2, so the unmatched test item information is found. The feedback unit 340 can record and feed back the time frame 1 and the specified information.

[0049] In this way, the error aspects of the tested perceptual visual recognition algorithm ALO can be fed back to the tester more quickly and deeply, so that the tester can easily find the cause of the error through the feedback time frame and specified information (for example, the collected pictures), and thus make timely and targeted repairs to improve testing efficiency.

[0050] In one embodiment, the sensor data set CDS may also include scene information. As described above, the sensor data set CDS may be pre-collected on the road surface of the mining area. Optionally, a road surface environment may be built indoors to collect the sensor data set CDS. In this embodiment, multiple sensor data sets CDS may be collected in different multiple scenes. As an example, multiple scenes may include scenes based on time changes (e.g., daytime, nighttime), scenes based on light changes (e.g., backlight, front light), scenes based on distance changes, scenes based on object state changes (e.g., dynamic (horizontal, vertical, circling, etc.) and static (horizontal, vertical, different angles, etc.)), scenes based on object type changes (e.g., arranging different types of obstacles, such as people, cones, stones, mining trucks, water trucks, pickup trucks, SUVs, excavators, bulldozers, shovels, rollers, and graders), and at least one of the scenes based on object quantity changes. It should be noted that the above scenes may be used alone or in combination, that is, various different scenes may be set at will based on the above-mentioned change factors, as long as the test requirements can be met. According to an embodiment of the present invention, each scene may be named or numbered to form scene information, and the formed scene information may be included in the sensor data set CDS.

[0051] Further, the publishing module 10 may pre-store multiple sensor data sets CDS collected in multiple scenarios, and be configured to input the multiple sensor data sets CDS into the perceptual visual recognition algorithm ALO to be tested, respectively, to obtain test results TR for multiple scenarios. As an example, in the process of testing the perceptual visual recognition algorithm ALO, the publishing module 10 may continuously input multiple sensor data sets CDS in multiple scenarios into the perceptual visual recognition algorithm ALO, to continuously output multiple test results TR.

[0052] In one embodiment, the reference recognition result RRE may include scene information. The comparison unit 330 may also be configured to obtain a reference recognition result RRE with the same scene information based on the scene information in the input sensor data set CDS, so as to compare the final recognition result FRE with the reference recognition result RRE based on the same scene. As an example, the publishing module 10 may continuously input multiple sensor data sets CDS into the perceptual visual recognition algorithm ALO to calculate multiple final recognition results FRE, and make each final recognition result FRE include corresponding scene information. The publishing module 10 also continuously sends multiple reference recognition results RRE to the comparison unit 330. The comparison unit 330 may extract the scene information included in the final recognition result FRE to be compared, and search for a reference recognition result RRE with the same scene information from multiple reference recognition results RRE based on the extracted scene information, thereby comparing the final recognition result FRE with the searched reference recognition result RRE.

[0053] In this way, algorithm testing in multiple scenarios can be achieved, thereby increasing the test dimension to improve the comprehensiveness, accuracy and stability of the test results. By using the simulation test method according to an embodiment of the present invention, algorithm testing in multiple scenarios can be easily performed without the need to use more manpower and equipment for field testing. In addition, since sensor data in multiple scenarios is collected in advance, test results with greater dimensional coverage can be completed in a shorter time, thereby improving the efficiency and accuracy of the test and accelerating the improvement progress of the perceptual visual recognition algorithm ALO.

[0054] In one embodiment, the comparison unit 330 may also be configured to perform statistical processing on the test results TR of multiple scenarios to obtain at least one of the detection success rate, the detection omission rate, and the false recognition rate of the test item. As an example, at least one of the detection success rate, the detection omission rate, and the false recognition rate of each test item may be calculated. As described above, in each test result TR, it may be recorded whether each test item is successfully detected, missed, or falsely detected. When multiple test results TR are output, the detection success rate of each test item may be calculated based on the recorded detection results of each test item (i.e., detection success, missed detection, or false detection). For example, for each test item, the detection success rate = the number of successful detections / total number × 100%, where the total number may be the total number of tests performed, i.e., it may be equal to the number of multiple sensor data sets CDS or the number of multiple scenarios input. However, the embodiments of the present invention are not limited thereto, and the detection success rate of the test item may be calculated in other suitable ways, as long as the probability or proportion of successful detection of the test item can be obtained. The detection omission rate and false recognition rate of each test item may be calculated in a similar manner.

[0055] In another example, at least one of the detection success rate, detection omission rate, and error recognition rate of the test item can also be counted according to the object type. For example, the detection success rate, detection omission rate, and error recognition rate of the human type, cone bucket type, stone type, mining truck type, sprinkler type, pickup type, SUV type, excavator type, bulldozer type, forklift type, roller type, and road grader type can be calculated. In another example, at least one of the detection success rate, detection omission rate, and error recognition rate of the test item can also be counted according to the scene characteristics. For example, the detection success rate, detection omission rate, and error recognition rate of the backlit scene, the frontlit scene, the night scene, the distance of 5 meters, the distance of 10 meters, the distance of 30 meters, the distance of 50 meters, the distance of 55 meters, all obstacles dynamic scenes, and all obstacles static scenes can be calculated. Additionally or optionally, the detection success rate, detection omission rate, and error recognition rate of the overall data can be calculated.

[0056] In this way, the calculation success rate of the perceptual visual recognition algorithm ALO under multiple dimensions can be calculated to reflect the performance of the perceptual visual recognition algorithm ALO at least in terms of the success rate. However, the embodiments of the present invention are not limited to this, and other data statistical processing can be performed on the test results TR of multiple scenes according to actual needs to reflect the performance of the perceptual visual recognition algorithm ALO.

[0057] Figure 5 A flow chart of a method for testing a perception visual recognition algorithm for an unmanned mining vehicle using an embodiment of the present invention is shown.

[0058] like Figure 5 As shown, in step S501, the sensor data set CDS is input into the perceptual visual recognition algorithm ALO to be tested to output the algorithm calculation result ARE.

[0059] In step S502, data processing is performed on the algorithm calculation result ARE to obtain object recognition information OED, and statistical processing is performed on the object recognition information OED to obtain a final recognition result FRE. The final recognition result FRE includes output results of multiple test items.

[0060] In step S503, the final recognition result FRE is compared with the reference recognition result RRE to obtain a test result TR.

[0061] Advantageously, by using the testing system and method for the perception visual recognition algorithm of the unmanned mining vehicle according to the embodiment of the present invention, the problems of high field testing cost, inability to accurately control field testing conditions, low efficiency, and potential safety hazards in the existing testing process are solved. In addition, outdoor field deployment is not required, and the occupation of operating resources and costs such as excavators, wide-body vehicles, and command vehicles is saved, thereby making the test coverage complete, the retest cost low, and improving the test efficiency.

[0062] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0063] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A testing method for a perception visual recognition algorithm for an unmanned mining vehicle, characterized in that: The test method includes: Inputting a pre-stored sensor data set into the perceptual visual recognition algorithm to be tested to output an algorithm calculation result; Performing data processing on the algorithm calculation result to obtain object recognition information, and performing statistical processing on the object recognition information to obtain a final recognition result, wherein the final recognition result includes output results of multiple test items; and The final recognition result is compared with a pre-stored reference recognition result to obtain a test result of the perceptual visual recognition algorithm.

2. The testing method according to claim 1, characterized in that: The sensor data set is acquired by at least one visual sensor used for the unmanned mining vehicle at multiple time frames, so that the sensor data set includes multiple time frames and sensor data corresponding to each time frame; as well as The perceptual visual recognition algorithm performs calculations on the sensor data of each time frame to obtain the algorithm calculation results.

3. The testing method according to claim 2, characterized in that: Performing data processing on the algorithm calculation results includes: performing format conversion on the algorithm calculation results, extracting the identified objects from the converted algorithm calculation results, and obtaining relevant information of the identified objects as the multiple test items, and storing the multiple test items in corresponding time frames to form the object recognition information.

4. The testing method according to claim 3, characterized in that: Performing statistical processing on the object recognition information includes: for each test item, performing proportion statistics in all time frames, so as to take the acquired information with the largest proportion as the output result of the corresponding test item, and record it in the final recognition result.

5. The testing method according to claim 4, characterized in that: The reference recognition result includes reference results of multiple test items; as well as Comparing the output results of the multiple test items with the reference results of the multiple test items includes: matching and comparing the output results with the reference results for each test item respectively to obtain test items that have been successfully detected, test items that have failed to be detected, test items that have been missed, and test items that have been incorrectly identified and recording them in the test results, and calculating the success rate based on the successfully identified test items.

6. The testing method according to claim 5, characterized in that: The test method further includes: determining a time frame of an output result including a detection failure in the object recognition information based on the reference recognition result, and recording and feeding back the determined time frame and designated information associated with the determined time frame.

7. The testing method according to claim 1, characterized in that: The sensor data set also includes scene information, and The testing method further includes: inputting a plurality of sensor data sets collected in a plurality of different scenes into the perceptual visual recognition algorithm to be tested to obtain the test results for the different scenes.

8. The testing method according to claim 7, characterized in that: The reference recognition result includes scene information, and Comparing the final recognition result with the pre-stored reference recognition result also includes: obtaining a reference recognition result with the same scene information according to the scene information in the input sensor data set, so as to compare the final recognition result with the reference recognition result based on the same scene.

9. The testing method according to claim 8, characterized in that: The test method also includes: The test results of the multiple scenarios are statistically processed to obtain at least one of a detection success rate, a detection omission rate, and an error recognition rate of the test items.

10. The testing method according to any one of claims 7 to 9, characterized in that: The multiple scenes include at least one of a scene based on time changes, a scene based on light changes, a scene based on distance changes, a scene based on object state changes, a scene based on object type changes, and a scene based on object quantity changes.

11. A testing system for the perception and visual recognition algorithm of an unmanned mining vehicle, characterized in that: The test system performs the test method according to any one of claims 1 to 10, and the test system comprises: a publishing module, the publishing module being configured to input a sensor data set into the perceptual visual recognition algorithm to be tested, the sensor data set being acquired by at least one visual sensor for the unmanned mining vehicle at multiple time frames and pre-stored in the publishing module; an algorithm execution module, the algorithm execution module being configured to cause the perceptual visual recognition algorithm to perform calculations according to the input sensor data set to output algorithm calculation results; and A subscription module is configured to perform data processing on the algorithm calculation result to obtain object recognition information, and perform statistical processing on the object recognition information to obtain a final recognition result, wherein the final recognition result includes output results of multiple test items.

12. The test system according to claim 11, characterized in that: The subscription module includes a data processing unit, which is configured to perform format conversion on the algorithm calculation results, extract the identified objects from the converted algorithm calculation results, and obtain relevant information of the identified objects as the multiple test items, and store the multiple test items in corresponding time frames to form the object identification information.

13. The test system according to claim 12, characterized in that: The subscription module further includes a data statistics unit, which is configured to perform proportion statistics in all time frames for each test item, so as to use the acquired information with the largest proportion as the output result of the corresponding test item.

14. The test system according to claim 13, characterized in that: The publishing module is further configured to send pre-stored reference recognition results to the subscription module, wherein the reference recognition results include reference results of multiple test items; as well as The subscription module also includes a comparison unit, which is configured to match and compare the output result with the reference result for each test item respectively, so as to obtain the test items that have been successfully detected, the test items that have failed to be detected, the test items that have been missed, and the test items that have been incorrectly identified and record them in the test result, and calculate the success rate based on the successfully identified test items.

15. The test system according to claim 14, characterized in that: The subscription module also includes a feedback unit, which is configured to determine a time frame of an output result including a detection failure in the object recognition information based on the reference recognition result, and record and feedback the determined time frame and specified information associated with the determined time frame. 16 . A computer program product, wherein the computer program product stores computer instructions, wherein the computer instructions are used to enable a computer to execute the testing method according to any one of claims 1 to 10.

Images