Method, system and program product for testing algorithms for unmanned mine vehicles

Abstract

The application provides a test method, system and program product of an algorithm for an unmanned mine car, and relates to the field of algorithm testing. The test method of a perception visual recognition algorithm for an unmanned mine car comprises the following steps: inputting a pre-stored sensor data set into a perception visual recognition algorithm to be tested to output an algorithm calculation result; performing data processing on the algorithm calculation result to obtain object recognition information, and performing statistical processing on the object recognition information to obtain a final recognition result, wherein the final recognition result comprises output results of multiple test items; and comparing the final recognition result with a pre-stored reference recognition result to obtain a test result of the perception visual recognition algorithm. The method and system of the application can avoid on-site manual testing.

Description

TECHNICAL FIELD

[0001] The present application relates to a test method, a test system and a computer program product for a perception visual recognition algorithm of an unmanned mine car. BACKGROUND

[0002] In the development and application of algorithm software, testing is an important link to ensure the quality and stability of the algorithm. For the perception visual recognition algorithm applied to the unmanned mine car, the traditional way is to conduct on-site manual testing. However, although on-site manual testing is accurate in some aspects, such testing requires a large amount of manpower, time and effort, and errors and omissions can still not be avoided. In addition, during a long testing process, personnel can make inaccurate judgments or miss testing due to fatigue. This not only wastes time and resources, but also has an indelible impact on the entire testing process.

[0003] On the other hand, due to the need for a large amount of human and time resources for on-site manual testing, batch testing cannot be performed. This means that multiple samples cannot be tested simultaneously, resulting in a prolonged testing period and limiting the production and listing time of the product. Therefore, it is necessary to introduce advanced automated testing methods to effectively solve the challenges faced by on-site testing. SUMMARY

[0004] The purpose of the present application is to provide a test method, a test system and a computer program product for a perception visual recognition algorithm of an unmanned mine car, which can realize batch testing in multiple scenarios and has low testing cost, high testing efficiency and good testing stability.

[0005] According to an aspect of the present application, a test method for a perception visual recognition algorithm of an unmanned mine car is provided, the method comprising: inputting a pre-stored sensor data set into the perception visual recognition algorithm to be tested to output an algorithm calculation result; performing data processing on the algorithm calculation result to obtain object recognition information, and performing statistical processing on the object recognition information to obtain a final recognition result, the final recognition result including output results of multiple test items; and comparing the final recognition result with a pre-stored reference recognition result to obtain a test result of the perception visual recognition algorithm.

[0006] In an embodiment, the sensor data set is obtained by at least one visual sensor for the unmanned mine car at multiple time frames, so that the sensor data set includes multiple time frames and sensor data corresponding to each time frame; and the perception visual recognition algorithm performs calculation on the sensor data of each time frame to obtain the algorithm calculation result.

[0007] In an embodiment, the data processing on the algorithm calculation result comprises: performing format conversion on the algorithm calculation result, extracting the identified object from the converted algorithm calculation result, and obtaining relevant information of the identified object as the plurality of test items, and storing the plurality of test items under the corresponding time frame to form the object identification information.

[0008] In an embodiment, the statistical processing on the object identification information comprises: for each test item, performing proportion statistics among all time frames to obtain the maximum proportion rate of the obtained information as the output result of the corresponding test item, and recording in the final identification result.

[0009] In an embodiment, the reference identification result comprises reference results of a plurality of test items; and the comparison of the output results of the plurality of test items with the reference results of the plurality of test items comprises: respectively for each test item, matching and comparing the output result with the reference result to obtain test items of detection success, test items of detection failure, test items of detection omission, and test items of false identification and record in the test result, and calculating a success proportion rate based on the test items of detection success.

[0010] In an embodiment, the test method further comprises: based on the reference identification result, determining a time frame in the object identification information including an output result of detection failure, and recording and feeding back the determined time frame and specified information associated with the determined time frame.

[0011] In an embodiment, the sensor data set further comprises scene information, and the test method further comprises: inputting a plurality of sensor data sets respectively collected in different scenes to the perception visual recognition algorithm to be tested to obtain the test result for different scenes.

[0012] In an embodiment, the reference identification result comprises scene information, and the comparison of the final identification result with the pre-stored reference identification result further comprises: according to the scene information in the input sensor data set, obtaining a reference identification result with the same scene information to compare the final identification result with the reference identification result based on the same scene.

[0013] In an embodiment, the test results of the plurality of scenes are statistically processed to obtain at least one of a detection success rate, a detection omission rate, and a false identification rate of the test items.

[0014] In embodiments, the plurality of scenarios includes at least one of a time-varying scenario, a light-varying scenario, a distance-varying scenario, an object state-varying scenario, an object type-varying scenario, and an object number-varying scenario.

[0015] According to another aspect of the present application, there is provided a test system for a perception visual recognition algorithm of an unmanned mine car, the test system performing the test method according to an embodiment of the present application, the test system comprising: a publishing module configured to input a sensor data set into the perception visual recognition algorithm to be tested, the sensor data set being obtained by at least one visual sensor for the unmanned mine car under a plurality of time frames and being pre-stored in the publishing module; an algorithm execution module configured to cause the perception visual recognition algorithm to perform calculation according to the input sensor data set to output an algorithm calculation result; and a subscription module configured to perform data processing on the algorithm calculation result to obtain object recognition information, and perform statistical processing on the object recognition information to obtain a final recognition result, the final recognition result including output results of a plurality of test items.

[0016] In embodiments, the subscription module includes a data processing unit configured to perform format conversion on the algorithm calculation result, and extract the recognized object from the converted algorithm calculation result, and obtain related information of the recognized object as the plurality of test items, and store the plurality of test items under corresponding time frames to form the object recognition information.

[0017] In embodiments, the subscription module further includes a data statistical unit configured to, for each test item, perform proportion statistics among all time frames to obtain the information with the largest proportion rate as the output result of the corresponding test item.

[0018] In embodiments, the publishing module is further configured to send a pre-stored reference recognition result to the subscription module, the reference recognition result including reference results of a plurality of test items; and the subscription module further includes a comparison unit configured to, for each test item, respectively, match and compare the output result with the reference result to obtain and record in the test result a test item with successful detection, a test item with failed detection, a test item with missed detection, and a test item with false recognition, and calculate a success proportion rate based on the test item with successful detection.

[0019] In an embodiment, the subscription module further comprises a feedback unit configured to determine, based on the reference recognition result, a time frame in the object recognition information that includes an output result of detection failure, and record and feedback the determined time frame and specified information associated with the determined time frame.

[0020] According to yet another aspect of the present application, there is provided a computer program product storing computer instructions for causing a computer to execute a method according to an embodiment of the present application.

[0021] It should be understood that nothing in this section is intended to limit the scope of the embodiments of the present application. Other aspects of the embodiments of the present application will become apparent from the following description, which, taken in conjunction with the accompanying drawings, discloses a preferred embodiment of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0022] The accompanying drawings are included to provide a further understanding of the present application, and are incorporated in and constitute a part of this specification, illustrate embodiments of the present application, and together with the description serve to explain the principles of the present application.

[0023] Figure 1 A schematic diagram of a test system for a perception visual recognition algorithm for an unmanned mine truck according to an embodiment of the present application is shown.

[0024] Figure 2 A schematic diagram of a subscription module according to an embodiment of the present application is shown.

[0025] Figure 3 An example of object recognition information according to an embodiment of the present application is shown.

[0026] Figure 4 An example of a final recognition result and a reference recognition result according to an embodiment of the present application is shown.

[0027] Figure 5 A flowchart of a test method for a perception visual recognition algorithm for an unmanned mine truck according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0028] In order to more clearly illustrate the objectives, technical solutions and merits of the present application, the embodiments of the present application will be described in detail hereinafter with reference to the accompanying drawings. It should be understood that the following description of the embodiments is intended to explain and describe the general inventive concept, but should not be construed as limiting the present application. In the specification and drawings, the same or similar reference signs refer to the same or similar components or members. For the sake of clarity, the drawings are not necessarily drawn to scale and some well-known components and structures can be omitted from the drawings.

[0029] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms “first,” “second,” and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. The word “a” or “an” does not exclude a plurality. Terms such as “comprising” or “including” mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as “connected” or “linked” are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as “upper,” “lower,” “left,” “right,” “top,” or “bottom,” etc., are used only to indicate relative positional relationships, which may change accordingly when the absolute position of the described object changes. When an element such as a layer, film, region, or substrate is referred to as being “above” or “below” another element, the element may be “directly” located “above” or “below” the other element, or there may be intermediate elements present.

[0030] Figure 1 A schematic diagram of a test system 1000 for a perception-visual recognition algorithm for an unmanned mining truck according to an embodiment of the present invention is shown. Figure 1 As shown, the test system 1000 may include a publishing module 10, an algorithm execution module 20, and a subscription module 30.

[0031] The publishing module 10 can pre-store a sensor data set (CDS) and is configured to input the pre-stored sensor data set CDS into the algorithm execution module 20. In this document, "sensor data set CDS" can be understood as a data set obtained by at least one visual sensor used in an unmanned mining truck in a mining scene. For example, a mining scene and location can be pre-specified, and then the unmanned mining truck can be driven to that location and at least one visual sensor on the unmanned mining truck can be used to collect data on the mining scene, thereby obtaining a sensor data set CDS corresponding to that mining scene. According to embodiments of the present invention, data can be collected in multiple scenes in advance to obtain multiple sensor data sets CDS. Then, all the collected sensor data sets CDS can be stored in the publishing module 10 for use in subsequent algorithm testing.

[0032] According to embodiments of the present invention, the sensor data set (CDS) may include sensor data acquired by at least one vision sensor of the unmanned mining vehicle at multiple time frames. For example, during data acquisition, at least one vision sensor of the unmanned mining vehicle may take pictures continuously for multiple time frames to obtain sensor data corresponding to each time frame. Therefore, the sensor data set (CDS) may include multiple time frames and sensor data corresponding to each time frame. As an example, each time frame may be 1 millisecond (ms).

[0033] Algorithm execution module 20 can be configured to receive a perceptual visual recognition algorithm (ALO) to be tested. In this document, "perceptual visual recognition algorithm (ALO)" can be understood as an algorithm designed for a control system suitable for unmanned mining trucks, capable of processing data collected by at least one visual sensor of the unmanned mining truck to obtain information about the surrounding roads in the mining area, thereby helping the unmanned mining truck avoid and bypass obstacles. According to embodiments of the present invention, algorithm execution module 20 can receive multiple perceptual visual recognition algorithm (ALOs) and optionally delete or replace the received ALOs. For example, after testing a perceptual visual recognition algorithm (ALO), if the test fails and adjustment is required, the adjusted algorithm can be updated and input into algorithm execution module 20 for a new test. As another example, after updating the perceptual visual recognition algorithm (ALO) to suit a specific usage scenario or requirement, it can also be directly input into algorithm execution module 20 for testing.

[0034] As an example, during testing, the perceptual visual recognition algorithm ALO to be tested can first be uploaded to the algorithm execution module 20, and then the publishing module 10 can input the sensor data set CDS into the algorithm execution module 20. According to an embodiment of the present invention, the algorithm execution module 20 can be configured to cause the perceptual visual recognition algorithm ALO to perform calculations based on the input sensor data set CDS, so as to output the algorithm calculation result ARE. As an example, as described above, the sensor data set CDS includes multiple sensor data corresponding to multiple time frames. The perceptual visual recognition algorithm ALO can perform calculations separately for the sensor data in each time frame, thereby obtaining multiple calculation results corresponding to multiple time frames, and thus forming the algorithm calculation result ARE.

[0035] The subscription module 30 can be configured to perform data processing and statistical processing on the algorithm calculation result ARE to obtain the final recognition result FRE. The publishing module 10 can also be configured to pre-store a reference recognition result RRE and send the reference recognition result RRE to the subscription module 30, so that the test result of the perceptual visual recognition algorithm ALO can be obtained by comparing the final recognition result FRE with the reference recognition result RRE.

[0036] In this way, the ALO (Alternating Locator) visual recognition algorithm can be tested without visiting the mining site, thus solving the technical problems of high cost and low efficiency associated with on-site testing. Furthermore, since on-site testing is unnecessary, it eliminates the need for significant manpower and time resources, enabling batch testing of multiple samples simultaneously without extending the testing cycle. This improves the effectiveness and accuracy of the testing.

[0037] Figure 2 A schematic diagram of a subscription module 30 according to an embodiment of the present invention is shown. Figure 2 As shown, the subscription module 30 includes a data processing unit 310, a data statistics unit 320, a comparison unit 330, and a feedback unit 340.

[0038] According to an embodiment of the present invention, the data processing unit 310 can perform data processing on the algorithm calculation result ARE to obtain object recognition information OED. The data processing unit 310 can be configured to perform format conversion on the algorithm calculation result ARE to extract the identified object from the converted algorithm calculation result ARE, and obtain relevant information of the identified object as multiple test items. These multiple test items are stored in corresponding time frames to form the object recognition information OED. As an example, after the perceptual visual recognition algorithm ALO performs calculation, the objects existing in the collected scene (or at least the objects that the unmanned mining truck must pay attention to during operation) and their relevant information (e.g., object type, distance from the unmanned mining truck, size, etc.) can be calculated and included in the algorithm calculation result ARE. However, since the perceptual visual recognition algorithm ALO is configured to work with the control system of the unmanned mining truck, the format of the algorithm calculation result ARE output from the perceptual visual recognition algorithm ALO may not be suitable or convenient for the test system 1000 to perform subsequent processing. The data processing unit 310 can perform format conversion on the algorithm calculation result ARE to convert it into a data format suitable for subsequent processing. Then, the data processing unit 310 can determine and extract one or more identified objects from the algorithm calculation result ARE after format conversion, and obtain information related to each object. In this way, through the format conversion and extraction process of the data processing unit 310, the identified objects and their related information can be represented in the desired format or form, thereby facilitating subsequent statistical processes.

[0039] Figure 3 An example of Object Identification Information (OED) according to an embodiment of the present invention is shown. As an example, Figure 3 The illustration shows the extraction and identification of three objects (i.e., object 1-object 3) from the algorithmic calculation result ARE based on three time frames (i.e., time frame 1-time frame 3), and the acquisition of three pieces of information (e.g., type, size, distance) for each object as test items (i.e., T1-T9). However, this is merely an example, and embodiments of the present invention are not limited thereto.

[0040] like Figure 3As shown, the type information of object 1 can be stored as test item T1, the distance information of object 1 can be stored as test item T2, the size information of object 1 can be stored as test item T3, the type information of object 2 can be stored as test item T4, the distance information of object 2 can be stored as test item T5, the size information of object 2 can be stored as test item T6, the type information of object 3 can be stored as test item T7, the distance information of object 3 can be stored as test item T8, and the size information of object 3 can be stored as test item T9. As mentioned above, the algorithm calculation result ARE includes the calculation result corresponding to each time frame, and therefore includes object-related information under each time frame, that is, the information of each test item T1-T9 under each time frame. After the data processing unit 310 obtains this information from the algorithm calculation result ARE, it can store the obtained information under the corresponding time frame to form object recognition information OED. For example, as Figure 3 As shown, for test item T1, it is identified as a person type in time frames 1-3; for test item T2, it is identified as distance L1 in time frames 1-2, but as distance L2 in time frame 3; for test item T3, it is identified as size S1 in time frames 1-3. Similarly, in this way, the acquisition information of each of test items T4-T9 in time frames 1-3 (e.g., size S3, S4, etc.) can be clearly known, thus facilitating subsequent statistical processing. However, it should be noted that the relevant information of the identified objects is not limited to type, size, and distance, and can include more other information depending on the nature of the algorithm and actual needs.

[0041] The data statistics unit 320 can perform statistical processing on the object recognition information (OED). The data statistics unit 320 can be configured to perform percentage statistics across all time frames for each test item, so that the information with the largest percentage is used as the output result of the corresponding test item and recorded in the final recognition result (FRE).

[0042] Figure 4 Examples of the final identification result (FRE) and the reference identification result (RRE) according to embodiments of the present invention are shown. As an example, such as... Figures 3-4 As shown, in the formation Figure 3After obtaining the object recognition information (OED), the data statistics unit 320 performs percentage statistics processing for each test item T1-T9 to determine the information with the highest percentage for each test item. For example, for test item T1, since it is identified as a human type in time frames 1-3, the information with the highest percentage is human type. For test item T2, since it is identified as distance L1 in two time frames (i.e., time frame 1 and time frame 2) but only as distance L2 in one time frame (i.e., time frame 3), the information with the highest percentage is distance L1. For test item T3, since it is identified as size S1 in all time frames 1-3, the information with the highest percentage is size S1. Similarly, the most significant information for test item T4 is the type of stone; for test item T5, it's distance L3; for test item T6, it's size S4; for test item T7, it's minecart type; and for test item T8, it's distance L5 and size S5. Therefore, the determined most significant information for each test item can be used as the output result for that test item, thus forming the final recognition result FRE. Therefore, as... Figure 4 As shown, the output results of test items T1-T9 in the final recognition result FRE are: human type, L1, S1, stone type, L3, S4, minecart type, L5, S5.

[0043] However, it should be noted that the above is merely an example, and only 3 time frames and 9 test items are shown for ease of understanding. In practical applications, more time frames and test items can be used. Furthermore, various suitable methods can be used to determine the information with the highest proportion in the proportion calculation. Besides counting the number of each type of information to determine the proportion, a threshold can also be set to determine the information with the highest proportion. For example, a threshold of 40% can be set, and when the number of time frames containing a certain type of information exceeds 40% of the total number of time frames, that information is considered to have the highest proportion. In other words, the method for calculating the information with the highest proportion can be predefined in any suitable way according to the actual situation, as long as the obtained result represents the testing trend of the corresponding identification item.

[0044] like Figure 4As shown, the Reference Recognition Result (RRE) can include reference results for multiple test items. As mentioned above, the Reference Recognition Result (RRE) is also pre-stored in the publishing module 10. In this document, "Reference Recognition Result (RRE)" can be understood as reference information acquired and recorded simultaneously when collecting the sensor data set (CDS), so that when the perception vision recognition algorithm (ALO) performs calculations based on the sensor data set (CDS), the corresponding reference information can be used for comparison to determine whether the test has passed. For example, the Reference Recognition Result (RRE) can include actual objects on the road surface and their related information, or it can include objects that the Reference Recognition Result (RRE) should recognize and their related information. As an example, Figure 4 The reference recognition result (RRE) shown includes reference results for 12 test items T1-T12 for 4 objects, and this reference recognition result (RRE) corresponds to the calculated... Figure 4 The final recognition result FRE uses the sensor data set CDS. In other words, the publishing module 10 can input the sensor data set CDS into the perceptual visual recognition algorithm ALO so that the subscription module 30 can calculate the final recognition result FRE (e.g., Figure 4 The final identification result (FRE) will be compared with the reference identification result (RRE) corresponding to the input sensor data set (CDS). Figure 4 The reference identification result (RRE) is sent to the subscription module 30, thereby comparing the reference identification result RRE with the final identification result FRE to output the test result TR.

[0045] The comparison unit 330 can perform a comparison between the reference recognition result (RRE) and the final recognition result (FRE). The comparison unit 330 can be configured to match the output result with the reference result for each test item separately, to obtain successfully detected test items, failed detected test items, missed detected test items, and incorrectly identified test items, and record them in the test result (TR). As an example, such as... Figure 4 As shown, one can first match the identified objects in the final recognition result FRE with the reference objects in the reference recognition result RRE based on object features, thereby matching the reference results and output results of each test item one by one. From Figure 4 As can be seen, the reference recognition result RRE includes 4 objects. Therefore, by comparison, it can be determined that object 4 and the corresponding test items T10-T12 (the reference recognition result RRE is for the stone, L7, and S7) were missed. Thus, test items T10-T12 can be recorded as missed test items in the test result TR. In another embodiment, all existing objects may be identified, but not all relevant information may be output. For example, object 4 may be identified, but test item T12 may not be output. In this case, test item T12, which detects missed test items, can be recorded in the test result TR.

[0046] For test items T1-T9, the output results can be matched and compared with the reference results respectively. For example, as shown... Figure 3 As shown, the output of test item T2 is L1, but the reference result is L2. Therefore, it can be determined that test item T12 failed to be detected, and can be recorded in the test result TR as a failed test item. On the other hand, for test items T1 and T3-T9, the output results match the reference results, so it can be determined that test items T1 and T3-T9 were successfully detected, and can be recorded in the test result TR as successfully detected test items. In another embodiment, there may be objects or test items included in the final recognition result FRE, but not included in the reference recognition result RRE. In this case, it can be determined that such test items or the test items corresponding to such objects are falsely detected, and can be recorded in the test result TR as falsely identified test items.

[0047] In this way, the test result TR can record successfully detected test items, failed test items, missed test items, and incorrectly identified test items. This allows testers to clearly and quickly understand the errors and performance of the tested algorithm, and thus make targeted improvements to the algorithm. Furthermore, the comparison unit 330 can also be configured to calculate the success rate based on the test result TR. For example, the success rate = number of successfully detected test items / total number of test items included in the reference recognition result RRE × 100%.

[0048] The subscription module 30 also includes a feedback unit 340. The feedback unit 340 can be configured to determine the time frame of the object recognition information (OED), including the output result of detection failure, based on the reference recognition result (RRE), and record and feed back the determined time frame along with the specified information associated with it. As an example, the feedback unit 340 can match and compare the information of each test item in each time frame with the information of each test item in the reference recognition result (RRE). When a mismatch is found, the feedback unit determines the time frame containing that test item information and simultaneously obtains the specified information associated with that time frame, such as the image captured in that time frame and the specific time information corresponding to that time frame. Then, the time frame and the specified information are recorded and fed back to the tester. For example, Figure 5 The information for test item T2 in time frame 1 is L1, but the information for test item T2 in the reference recognition result RRE is L2, thus a mismatch in test item information was found. Feedback unit 340 can record and feed back time frame 1 and the specified information.

[0049] In this way, the errors of the ALO (Automatic Visual Recognition) algorithm being tested can be fed back to the testers more quickly and in greater depth. This allows the testers to easily find the cause of the error by using the time frame and specified information (e.g., the captured image) provided by the feedback, and thus make timely and targeted fixes to improve testing efficiency.

[0050] In one embodiment, the sensor data set CDS may further include scene information. As mentioned above, the sensor data set CDS can be pre-collected on the road surface in the mining area. Optionally, a road environment can also be constructed indoors to collect the sensor data set CDS. In this embodiment, multiple sensor data sets CDS can be collected under different scenarios. As an example, multiple scenarios may include at least one of the following: scenarios based on time changes (e.g., daytime, nighttime), scenarios based on light changes (e.g., backlighting, front lighting), scenarios based on distance changes, scenarios based on object state changes (e.g., dynamic (lateral, longitudinal, rotating, etc.) and static (lateral, longitudinal, different angles, etc.)), scenarios based on object type changes (e.g., different types of obstacles such as people, cones, rocks, mining trucks, water trucks, pickup trucks, SUVs, excavators, bulldozers, loaders, road rollers, and graders), and scenarios based on changes in the number of objects. It should be noted that the above scenarios can be used individually or in combination; that is, various different scenarios can be arbitrarily set based on the above-mentioned changing factors, as long as the testing requirements are met. According to embodiments of the present invention, each scene can be named or numbered to form scene information, and the formed scene information can be included in the sensor data set CDS.

[0051] Furthermore, the publishing module 10 can pre-store multiple sensor data sets (CDS) collected in multiple scenarios, and is configured to input the multiple sensor data sets (CDS) into the perception vision recognition algorithm (ALO) to be tested, respectively, to obtain test results (TR) for multiple scenarios. As an example, during the testing of the perception vision recognition algorithm (ALO), the publishing module 10 can continuously input multiple sensor data sets (CDS) from multiple scenarios into the perception vision recognition algorithm (ALO) to continuously output multiple test results (TR).

[0052] In one embodiment, the reference recognition result RRE may include scene information. The comparison unit 330 may also be configured to obtain reference recognition results RREs with the same scene information based on the scene information in the input sensor data set CDS, so as to compare the final recognition result FRE with the reference recognition result RRE based on the same scene. As an example, the publishing module 10 may continuously input multiple sensor data sets CDS into the perceptual vision recognition algorithm ALO to calculate multiple final recognition results FREs, and make each final recognition result FRE include corresponding scene information. The publishing module 10 also continuously sends multiple reference recognition results RREs to the comparison unit 330. The comparison unit 330 may extract the scene information included in the final recognition result FRE to be compared, and search for reference recognition result RREs with the same scene information from among the multiple reference recognition result RREs based on the extracted scene information, thereby comparing the final recognition result FRE with the searched reference recognition result RRE.

[0053] In this way, algorithm testing can be performed in multiple scenarios, thereby increasing the testing dimensions and improving the comprehensiveness, accuracy, and stability of the test results. By using the simulation testing method according to embodiments of the present invention, algorithm testing in multiple scenarios can be easily conducted without requiring significant manpower and equipment for on-site testing. Furthermore, because sensor data from multiple scenarios is collected in advance, test results with greater dimensional coverage can be completed in a shorter time, thereby improving testing efficiency and accuracy and accelerating the development of the visual recognition algorithm ALO.

[0054] In one embodiment, the comparison unit 330 can also be configured to perform statistical processing on the test results TR of multiple scenarios to obtain at least one of the detection success rate, detection omission rate, and false recognition rate of the test item. As an example, at least one of the detection success rate, detection omission rate, and false recognition rate can be calculated for each test item. As described above, each test result TR can record whether each test item was successfully detected, missed, or falsely detected. When multiple test results TR are output, the detection success rate of each test item can be calculated based on the recorded detection result of each test item (i.e., successful detection, missed detection, or false detection). For example, for each test item, the detection success rate = number of successful detections / total number × 100%, where the total number can be the total number of tests performed, i.e., equal to the number of multiple sensor data sets (CDS) or the number of multiple scenarios. However, embodiments of the present invention are not limited to this; the detection success rate of the test item can be calculated in other suitable ways, as long as the detection success probability or percentage of the test item can be obtained. The detection omission rate and false recognition rate of each test item can be calculated in a similar manner.

[0055] In another example, at least one of the following can be calculated based on the object type: detection success rate, detection omission rate, and false recognition rate. For example, the detection success rate, detection omission rate, and false recognition rate can be calculated for human type, cone type, rock type, mining truck type, sprinkler truck type, pickup truck type, SUV type, excavator type, bulldozer type, loader type, road roller type, and grader type. In yet another example, at least one of the following can be calculated based on scene characteristics: detection success rate, detection omission rate, and false recognition rate. For example, the detection success rate, detection omission rate, and false recognition rate can be calculated for backlit scenes, frontlit scenes, nighttime scenes, distances of 5 meters, 10 meters, 30 meters, 50 meters, 55 meters, all obstacle dynamic scenes, and all obstacle static scenes. Additionally or optionally, the detection success rate, detection omission rate, and false recognition rate of the overall data can be calculated.

[0056] In this way, the computational success rate of the ALO (Automatic Locator) algorithm for perceptual visual recognition under multiple dimensions can be calculated, thus reflecting the performance of the ALO algorithm at least in terms of success rate. However, embodiments of the present invention are not limited to this; other data statistical processing can be performed on the test results TR (Test Results of Multiple Scenarios) for multiple scenarios according to actual needs to reflect the performance of the ALO algorithm.

[0057] Figure 5 A flowchart is shown of a test method using a perception vision recognition algorithm for an unmanned mining truck according to an embodiment of the present invention.

[0058] like ​ As shown, in step S501, the sensor data set CDS is input into the perceptual visual recognition algorithm ALO to be tested, so as to output the algorithm calculation result ARE.

[0059] In step S502, data processing is performed on the algorithm calculation result ARE to obtain object recognition information OED, and statistical processing is performed on the object recognition information OED to obtain the final recognition result FRE. The final recognition result FRE includes the output results of multiple test items.

[0060] In step S503, the final identification result FRE is compared with the reference identification result RRE to obtain the test result TR.

[0061] Advantageously, by using the testing system and method for a perception-visual recognition algorithm for unmanned mining trucks according to embodiments of the present invention, the problems of high on-site testing costs, inaccurate control of on-site testing conditions, low efficiency, and potential safety hazards in existing testing processes are solved. Furthermore, outdoor on-site deployment is unnecessary, saving the use of excavators, wide-body vehicles, command vehicles, and other operational resources and costs, thereby achieving complete testing coverage, low retesting costs, and improved testing efficiency.

[0062] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0063] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A testing method for a perception-visual recognition algorithm for an unmanned mining truck, wherein the perception-visual recognition algorithm is configured to process data collected by at least one visual sensor of the unmanned mining truck to obtain obstacle-based road conditions in the collected scene, characterized in that, The testing method includes: The pre-stored sensor data set is input into the perceptual visual recognition algorithm to be tested to output the algorithm calculation result; The algorithm calculation results are format converted, and the identified objects are determined from the converted algorithm calculation results as obstacles in the collected scene. At least one relevant information of the identified objects is obtained as multiple test items. The multiple test items are stored in the corresponding time frames to form object recognition information. The relevant information of the identified objects includes the type of the object, the distance to the unmanned mining truck, and the size of the object. Statistical processing is performed on the object recognition information to obtain a final recognition result, which includes at least one recognized object and output results of multiple test items corresponding to each recognized object; and The objects identified in the final identification result are matched with the reference objects in the pre-stored reference identification result to determine whether there are any missed or false detections. The output results of each test item of the successfully matched object are compared with the reference results to determine whether the detection is successful.

2. The test method according to claim 1, characterized in that, The sensor data set is acquired by at least one vision sensor used in the unmanned mining vehicle at multiple time frames, such that the sensor data set includes multiple time frames and sensor data corresponding to each time frame. as well as The perception and visual recognition algorithm performs calculations on the sensor data of each time frame to obtain the algorithm's calculation results.

3. The test method according to claim 1, characterized in that, Performing statistical processing on the object recognition information includes: for each test item, performing a percentage statistics across all time frames, so that the information with the largest percentage is taken as the output result of the corresponding test item and recorded in the final recognition result.

4. The test method according to claim 3, characterized in that, The reference identification results include reference results for multiple test items; as well as Comparing the output results of the multiple test items with the reference results of the multiple test items includes: matching and comparing the output results with the reference results for each test item to obtain the test items that were successfully detected, the test items that failed to be detected, the test items that were missed, and the test items that were incorrectly identified, and recording them in the test results; and calculating the success rate based on the successfully identified test items.

5. The test method according to claim 4, characterized in that, The testing method further includes: based on the reference recognition result, determining the time frame of the object recognition information that includes the output result of the detection failure, and recording and feeding back the determined time frame and the specified information associated with the determined time frame.

6. The test method according to claim 1, characterized in that, The sensor data set also includes scene information, and The testing method further includes: inputting the perceptual visual recognition algorithm to be tested with multiple sensor data sets collected in different scenarios to obtain test results for different scenarios.

7. The test method according to claim 6, characterized in that, The reference recognition result includes scene information, and Comparing the final recognition result with a pre-stored reference recognition result further includes: obtaining a reference recognition result with the same scene information based on the scene information in the input sensor data set, so as to compare the final recognition result with the reference recognition result based on the same scene.

8. The test method according to claim 7, characterized in that, The testing method also includes: The test results of the multiple scenarios are statistically processed to obtain at least one of the following: detection success rate, detection omission rate, and false recognition rate of the test item.

9. The test method according to any one of claims 6-8, characterized in that, The plurality of scenarios includes at least one of the following: scenarios based on time changes, scenarios based on light changes, scenarios based on distance changes, scenarios based on object state changes, scenarios based on object type changes, and scenarios based on object quantity changes.

10. A testing system for a perception and vision recognition algorithm used in unmanned mining trucks, characterized in that, The testing system executes the testing method according to any one of claims 1 to 9, the testing system comprising: The publishing module is configured to input a sensor data set into the perception vision recognition algorithm to be tested. The sensor data set is acquired by at least one vision sensor for the unmanned mining truck at multiple time frames and is pre-stored in the publishing module. An algorithm execution module, configured to cause the perceptual visual recognition algorithm to perform calculations based on the input sensor data set, and to output the algorithm calculation results; and The subscription module is configured to perform data processing on the algorithm calculation results to obtain object recognition information, and to perform statistical processing on the object recognition information to obtain a final recognition result, the final recognition result including the output results of multiple test items.

11. The testing system according to claim 10, characterized in that, The subscription module includes a data processing unit, which is configured to perform format conversion on the algorithm calculation results, extract the identified objects from the converted algorithm calculation results, obtain relevant information of the identified objects as the multiple test items, and store the multiple test items in the corresponding time frames to form the object recognition information.

12. The testing system according to claim 11, characterized in that, The subscription module also includes a data statistics unit, which is configured to perform percentage statistics for each test item across all time frames, so as to take the information with the largest percentage as the output result of the corresponding test item.

13. The testing system according to claim 12, characterized in that, The publishing module is also configured to send pre-stored reference identification results to the subscription module, the reference identification results including reference results for multiple test items; as well as The subscription module further includes a comparison unit, which is configured to match and compare the output result with the reference result for each test item to obtain the test items that were successfully detected, the test items that failed to be detected, the test items that were missed, and the test items that were incorrectly identified and record them in the test results, and to calculate the success rate based on the successfully identified test items.

14. The testing system according to claim 13, characterized in that, The subscription module further includes a feedback unit, which is configured to determine, based on the reference recognition result, a time frame in the object recognition information that includes the output result of the detection failure, and record and feed back the determined time frame and the specified information associated with the determined time frame.

15. A computer program product storing computer instructions for causing a computer to perform the test method according to any one of claims 1-9.

Images