Automated assessment method, apparatus, system and program product

Abstract

The application relates to the field of automation, in particular to an automatic evaluation method, device, system and program product. The method comprises the following steps: acquiring image data comprising a detection target and label information of the image data, and deploying a to-be-evaluated algorithm to a plurality of computing nodes; performing standardization processing on the image data and the label information to generate standard data; dividing the standard image data into a plurality of task fragments, and distributing the task fragments to the computing nodes for processing to obtain detection results; and determining an evaluation result of the to-be-evaluated algorithm according to the detection results and the standard label information. The method can effectively adapt to the evaluation requirements of different image data, is beneficial to improving evaluation accuracy and evaluation performance, simplifying the evaluation process, and improving evaluation efficiency.

Description

Technical Field

[0001] This application relates to the field of automation, and in particular to an automated evaluation method, apparatus, system, and program product. Background Technology

[0002] With the widespread application of humanoid robots in warehousing and logistics, industrial handling and other fields, box recognition, as a core perception task, directly affects the efficiency and reliability of robot operations due to the performance of its algorithm.

[0003] Currently, the evaluation of box recognition algorithms typically relies on manual comparison, which is cumbersome and prone to subjective errors. The evaluation dimensions are limited, usually focusing only on detection accuracy, making it difficult to comprehensively reflect the actual performance of the algorithm. The large-scale data processing capabilities are insufficient, making it impossible to efficiently complete parallel evaluation and quantification of computational performance when faced with massive amounts of real-world scenarios and synthetic data. The algorithm integration compatibility is poor, with different manufacturers' recognition algorithms having inconsistent input and output formats, resulting in repetitive and inefficient evaluation processes. Summary of the Invention

[0004] In view of this, embodiments of this application provide automated evaluation methods, apparatus, systems and program products to solve the problems of complex evaluation processes, low accuracy, low efficiency and poor compatibility in the prior art.

[0005] A first aspect of this application provides an automated evaluation method, the method comprising: Acquire image data including the detected target and the annotation information of the image data, and deploy the algorithm to be evaluated to multiple computing nodes; The image data is standardized to generate standard image data, and the annotation information is standardized to generate standard annotation information. The standard image data is divided into multiple task slices, and the task slices are assigned to the computing nodes for processing to obtain detection results. The detection results include at least two of the following: bounding box detection results, segmentation mask, pose estimation results, and runtime. The evaluation result of the algorithm to be evaluated is determined based on the detection results and the standard labeling information.

[0006] In conjunction with the first aspect, in a first possible implementation of the first aspect, the task is partitioned and distributed to the computing nodes for processing to obtain detection results, including at least two of the following: The task is divided into segments and distributed to the computing nodes for processing to obtain the first bounding box region output by the algorithm to be evaluated. Combined with the second bounding box region in the standard annotation information, the intersection-union ratio (IU) used to represent the detection result of the bounding box is determined. The task fragments are allocated to the computing nodes for processing to obtain the first set of pixels corresponding to the segmentation mask output by the algorithm to be evaluated. Combined with the second set of pixels corresponding to the segmentation mask in the standard annotation information, a similarity coefficient is determined to represent the detection result of the segmentation mask. The task is divided and distributed to the computing node for processing to obtain the first translation vector and the first rotation vector output by the algorithm to be evaluated. Combined with the second translation vector and the second rotation vector in the standard annotation information, a pose estimation index is determined to represent the detection result of pose estimation. The task is partitioned and distributed to the computing nodes for processing to obtain the computational performance index of the detection result output by the algorithm to be evaluated. The computational performance index includes at least one of throughput, average processing latency and stability index.

[0007] In conjunction with the first aspect, in a second possible implementation of the first aspect, before dividing the standard image data into multiple task slices, the method further includes at least one of the following: The standard annotation information is stored in structured data; Consistency verification of standard annotation information for the same detection target is performed through feature matching and geometric verification.

[0008] In conjunction with the second possible implementation of the first aspect, in the third possible implementation of the first aspect, the standard annotation information is stored in structured data, including: The structured data stores the correspondence between image ID, target category, bounding box coordinates, segmentation mask, pose parameters, annotation time, and annotation personnel in the standard annotation information; Consistency verification of standard annotation information for the same detection target is performed through feature matching and geometric verification, including: Through feature matching and geometric verification, the consistency of standard annotation information of the same detection target in multiple frames of images is verified, as well as the overlap of bounding boxes, similarity of segmentation bounding boxes, and rationality of pose changes of the same detection target in multiple adjacent frames of images are verified.

[0009] In conjunction with the first aspect, in a fourth possible implementation of the first aspect, the standard image data is divided into multiple task slices, including: Determine the feature type of the standard image data; Based on the number of computing nodes, the standard image data with the same feature type is allocated to each computing node, and the task fragments corresponding to each computing node are obtained based on the standard image data allocated to each computing node.

[0010] In conjunction with the first aspect, in the fifth possible implementation of the first aspect, the algorithm to be evaluated is deployed to multiple computing nodes, including: The algorithm to be evaluated is received through a standardized interface, and the deployment data of the algorithm to be evaluated is received, including resource configuration information and software environment information. Based on the resource configuration information and software environment information, the algorithm to be evaluated is deployed on at least one computing node.

[0011] In conjunction with the first aspect to the fifth possible implementation of the first aspect, in the sixth possible implementation of the first aspect, the image data is standardized to generate standard image data, including at least one of the following: The image dimensions in the image data are normalized to a predetermined resolution. Map the pixel values ​​in the image data to a preset pixel range; Filter out salt-and-pepper noise and Gaussian noise from the image data; The annotation information is standardized to generate standard annotation information, including at least one of the following: Convert the bounding box detection results into pixel coordinates; Convert the segmentation mask into a binary image format; The pose estimation result is represented as a translation vector and a rotation matrix.

[0012] A second aspect of this application provides an automated evaluation apparatus, the apparatus comprising: The data acquisition unit is used to acquire image data including the detected target and the annotation information of the image data, and to deploy the algorithm to be evaluated to multiple computing nodes; The standardization processing unit is used to standardize the image data to generate standard image data, and to standardize the annotation information to generate standard annotation information. The segmentation processing unit is used to divide the standard image data into multiple task segments and assign the task segments to the computing nodes for processing to obtain detection results. The detection results include at least two of the following: bounding box detection results, segmentation mask, pose estimation results, and runtime. The evaluation result determination unit is used to determine the evaluation result of the algorithm to be evaluated based on the detection result and the standard labeling information.

[0013] A third aspect of this application provides an automated evaluation system, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the automated evaluation system performs the method as described in any of the first aspects.

[0014] A fourth aspect of this application provides a computer program product that, when run on a computer, causes the computer to execute the methods described in the first aspect or its various implementations.

[0015] A fifth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method as described in any of the first aspects.

[0016] A sixth aspect of this application provides a chip for implementing the methods in the various implementations of the first aspect described above. Specifically, the chip includes a processor for calling and running a computer program from a memory, causing a device equipped with the chip to perform the methods as described in the first aspect or its various implementations.

[0017] The beneficial effects of this application embodiment compared with the prior art are as follows: This application embodiment acquires image data and annotation information, deploys the algorithm to be evaluated to multiple computing nodes, standardizes the image data to generate standard image data, and standardizes the annotation information to generate standard annotation information. This can effectively adapt to the evaluation requirements of different image data. The standard image data is divided into multiple task slices and distributed to each computing node, which can complete the evaluation more efficiently. The detection result is determined based on at least two of the bounding box detection result, segmentation mask, pose estimation result and runtime, which can more accurately reflect the actual performance of the algorithm. The standardization process can effectively adapt to the data access of different algorithms, which helps to simplify the evaluation process and improve the evaluation efficiency. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram illustrating an implementation scenario of an automated evaluation method provided in an embodiment of this application; Figure 2 This is a schematic diagram illustrating the implementation process of an automated evaluation method provided in an embodiment of this application; Figure 3 This is a schematic diagram illustrating the implementation process of preprocessing standard annotation information according to an embodiment of this application; Figure 4 This is a schematic diagram illustrating an implementation process for determining a detection result provided in an embodiment of this application; Figure 5 This is a schematic diagram illustrating the implementation process of an automated evaluation method provided in an embodiment of this application; Figure 6 This is a schematic diagram of an automated evaluation device provided in an embodiment of this application; Figure 7 This is a schematic diagram of an automated evaluation system provided in an embodiment of this application. Detailed Implementation

[0020] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0021] To illustrate the technical solution described in this application, specific embodiments are provided below.

[0022] With the widespread application of humanoid robot technology in warehousing and logistics, industrial handling and other fields, box recognition, as a core perception capability for achieving autonomous operation, directly determines the overall efficiency and reliability of the robot in performing tasks.

[0023] However, current evaluations of various box recognition algorithms generally suffer from the following pain points: the evaluation process relies heavily on manual comparison of results, which is cumbersome and easily influenced by subjective judgment; the evaluation criteria are too simplistic, often focusing only on detection accuracy while neglecting comprehensive considerations of segmentation accuracy, pose estimation error, and computational efficiency, failing to fully and accurately reflect the algorithm's overall performance; when faced with massive amounts of real-world and synthetic data, there is a lack of efficient large-scale data processing and parallel evaluation capabilities, making it difficult to effectively quantify the algorithm's computational performance; furthermore, the lack of unified standards in input and output formats among different vendors' algorithms leads to difficulties in integrating evaluation systems, repetitive processes, and overall low evaluation efficiency.

[0024] Figure 1This diagram illustrates an application scenario of an automated evaluation method provided in this application. The scenario includes an automated evaluation device and multiple computing nodes. The automated evaluation device performs initialization tasks, including receiving raw data, performing standardized preprocessing, and managing standard data (including standard image data and standard annotation information). Users can configure parameters, upload algorithms, and trigger evaluation tasks through the automated evaluation device. Based on the configured parameters, the automated evaluation device deploys the algorithm to be evaluated to each computing node, intelligently dividing large-scale evaluation data into multiple "task slices," and sending these slices to each computing node based on the real-time load of each node. Each computing node can independently run the algorithm and process its received task slices. The local computing resources (GPU) of the computing node can be used to execute the detection algorithm, and based on the standards issued by the automated evaluation device, preliminary calculations of various accuracy indicators and performance data are performed. Each computing node returns its local calculation results to the automated evaluation device. The automated evaluation device receives all returned results, performs aggregation and statistical analysis (such as calculating global averages, rankings, and conducting scene comparison analysis), and generates final visualization charts and a comprehensive evaluation report.

[0025] Figure 2 A schematic diagram illustrating the implementation process of an automated evaluation method provided in this application is described in detail below: In S201, image data including the detected target and the annotation information of the image data are acquired, and the algorithm to be evaluated is deployed to multiple computing nodes.

[0026] The detection target in this embodiment can be an object that needs to be identified and manipulated in scenarios such as warehousing and logistics, and industrial handling, such as boxes that a humanoid robot needs to move. Image data includes image-related data of the detection target. Image data can include data collected from real-world scenes, synthetic data, and adapted data from publicly available datasets. Data collected from real-world scenes can include image sequences taken by a humanoid robot in a warehouse, and synthetic data can be model images generated based on 3D model rendering.

[0027] The annotation information refers to the "ground truth" information of each detected target in the image data, used as the evaluation benchmark. This annotation information includes, but is not limited to, bounding box coordinates, binary images of the segmentation mask, pose parameters, etc. The algorithm to be evaluated refers to one or more object detection algorithms used for performance evaluation. The computing nodes are hardware servers or virtual machines with computing capabilities, used to execute the evaluation tasks in parallel to improve evaluation efficiency.

[0028] When deploying the algorithm to be evaluated to multiple computing nodes, a standardized interface can be used to receive the algorithm and its deployment data. The deployment data may include resource configuration information and software environment information. Based on the resource configuration and software environment information, the algorithm to be evaluated is deployed on at least one computing node.

[0029] The standardized interface is a predefined input / output specification for the automated evaluation system, including how to receive input data (such as image paths, parameters, etc.) and how to return output results (such as the format of bounding box lists, etc.). Data standard adaptation based on the standardized interface requires that the algorithm under evaluation can receive and correctly parse the standard input parameters defined by the system, and that the data output by the algorithm is in the system's unified standard format.

[0030] Deployment data includes resource configuration information and software environment information. Resource configuration information refers to the hardware resource specifications required to run the algorithm, including the required CPU, GPU, and memory resources. Software environment information refers to the software dependencies required to run the algorithm, including the operating system version, programming language interpreter version, deep learning framework and version, and a list of third-party libraries.

[0031] By defining a unified interface and encapsulating deployment requirements, the heterogeneity of different algorithms can be shielded, enabling different algorithms to be quickly integrated into the automated evaluation system and automatically complete the environment configuration and deployment on specified computing resources, greatly improving the convenience and scalability of the evaluation.

[0032] In S202, the image data is standardized to generate standard image data, and the annotation information is standardized to generate standard annotation information.

[0033] The standardization processing in this application includes image data standardization and annotation information standardization. Specifically, image data standardization generates standard image data, and annotation information standardization generates standard annotation information.

[0034] Before standardization, a validation process can be performed, and then the validated data can be processed. During validation, checks can be made to ensure the image file is not corrupted and that the annotation information is complete (including bounding boxes, segmentation masks, and pose parameters). Processing validated images can include at least one of the following: The image dimensions in the image data are normalized to a predetermined resolution. All images can be scaled to a uniform resolution while maintaining a predetermined aspect ratio, such as scaling to a resolution of 1920*1080.

[0035] The pixel values ​​in the image data are mapped to a preset pixel range. Pixel values ​​can be normalized to a predetermined pixel range, such as [0, 1].

[0036] Salt-and-pepper noise and Gaussian noise are filtered out from the image data. An adaptive median filter can be used to remove salt-and-pepper noise and Gaussian noise from the image.

[0037] When processing the verified annotation information, at least one of the following may be included: The bounding box detection results are converted into pixel coordinates. Annotation formats from different sources are converted to the system's internal standard format. For example, the bounding box representation format can be converted from (center point coordinates, length, width) to (top left vertex, bottom right vertex) format.

[0038] Convert the segmentation mask into a binary image format. For example, uniformly express the segmentation mask as a PNG binary image format.

[0039] The pose estimation result is represented as a translation vector and a rotation matrix. The translation vector can be represented as (tx, ty, tz), and the rotation matrix can be represented as a 3*3 matrix R.

[0040] This step, through automated verification and preprocessing, reduces human intervention, avoids distortion of evaluation results due to dirty or non-standard data, and ensures the accuracy of subsequent indicator calculations.

[0041] In S203, the standard image data is divided into multiple task slices, and the task slices are assigned to the computing nodes for processing to obtain detection results.

[0042] The detection results include at least two of the following: bounding box detection results, segmentation mask, pose estimation results, and runtime (which can be determined based on the start and end timestamps).

[0043] Task segmentation refers to dividing large-scale standard image data into independently processable subsets according to a certain strategy, such as based on data volume or data features. The detection result is the output of the algorithm under evaluation after processing the task segmentation, including but not limited to at least two of the following: bounding boxes, segmentation masks, pose estimation, and the runtime consumed by the algorithm. This allows for a more comprehensive and reliable evaluation of the algorithm based on two or more performance metrics other than accuracy.

[0044] When allocating tasks, a dynamic load balancing strategy can be adopted based on the real-time load of the computing nodes (including metrics such as CPU utilization and GPU memory usage) to evenly distribute multiple task shards across the nodes. After receiving a task shard, the computing node calls the deployed evaluation algorithm to perform inference calculations on the images within the shard and generate detection results.

[0045] By sharding massive amounts of data and scheduling them to multiple computing nodes for parallel execution, the overall efficiency of the evaluation can be significantly improved, and the status of resources and tasks can be monitored in real time, ensuring the stable operation of the evaluation process.

[0046] In this embodiment of the application, when standard image data is divided into multiple task fragments, the feature type of the standard image data can be determined. Based on the number of computing nodes, standard image data with the same feature type can be allocated to each computing node. Based on the standard image data allocated to each computing node, the task fragments corresponding to each computing node can be obtained.

[0047] The feature type is the inherent attribute or classification label of standard image data. It can include attributes such as different lighting conditions (natural light or artificial light), occlusion intensity (partial or complete occlusion), and target size.

[0048] By identifying data features and sharding them accordingly, it can be ensured that the data processed by each computing node is relatively balanced in terms of data type distribution, avoiding evaluation bias caused by uneven data distribution (for example, one node only processes type A scenarios, while another node processes all type B scenarios). This allows for a fairer evaluation of the comprehensive performance of the detection algorithm in different scenarios and makes full use of all computing resources.

[0049] After image data preprocessing, the system can assign feature labels to each image by reading its metadata or through lightweight analysis. For example, the data can be categorized into feature labels such as "natural light, unobstructed, large box" and "lit, partially occluded, medium box".

[0050] Based on the number of currently available computing nodes, data with the same characteristic type are distributed as evenly as possible to each computing node, so that each computing node is assigned a data subset, which can be used as a task shard.

[0051] Before dividing the standard image data into multiple task fragments, the embodiments of this application can further optimize the standard image data, specifically as follows: Figure 3 As shown, it includes: In S301, information is labeled using structured data storage standards.

[0052] The structured data in this application embodiment can be a database system that organizes, stores, and manages data in tabular form, thereby enabling the systematic and efficient storage and management of massive, multi-dimensional standard annotation information, facilitating rapid retrieval, batch updates, and data consistency maintenance, and providing a stable and reliable data source for subsequent evaluation.

[0053] When storing data in structured format, the structured data includes the image ID, target category, bounding box coordinates, segmentation mask, pose parameters, annotation time, and the correspondence between the annotation personnel and other information in the standard annotation information.

[0054] This mapping uses images as the basic unit, linking detailed information and annotation data of the detected targets within them. The image ID uniquely identifies a specific image after standardization. The target category refers to the type or class of the detected target in the image. For example, when the detected target is "box," the target category could include categories such as "cardboard box" or "turnaround box." The bounding box coordinates can be the coordinates of the two vertices of the bounding box in pixel coordinates, including the coordinates of the top-left and bottom-right vertices. The segmentation mask represents the precise pixel-level geometric information of the target contour and can be represented in a binary image format. For example, the pixel containing the target can be represented as 1, and background pixels as 0. The pose parameters describe the position and orientation information of the detected target in 3D space. They can be represented as translation vectors and rotation matrices in world coordinates. The annotation time records the timestamp of when the annotation information was created or last updated. The annotation personnel records the information of the personnel who performed this annotation work for quality control, accountability, and possible annotation consistency calibration.

[0055] In S302, the consistency of standard annotation information for the same detection target is verified through feature matching and geometric verification.

[0056] Feature matching refers to extracting and comparing visual features of the same target in different images, such as SIFT features and ORB features, to determine whether they are the same object.

[0057] Geometric verification refers to checking the logical rationality based on the geometric attributes of the target being detected, such as the position, size, and orientation of the target's bounding box.

[0058] By cross-validating the standard annotation information of the same target in consecutive frames or multi-view images, potentially contradictory annotation errors are identified and marked. For example, if the size or position of the same detected target changes abruptly in two adjacent frames, the annotation error is adjusted and corrected to improve the overall quality and reliability of the standard annotation data used as the evaluation benchmark.

[0059] By using feature matching and geometric verification, we can verify the consistency of standard annotation information of the same target in multiple frames of images, as well as the overlap of bounding boxes, similarity of segmented bounding boxes, and rationality of pose changes of the same target in multiple adjacent frames of images.

[0060] For the same detected target in different frames of a related video sequence, you can check whether the labeled target category, size, and other information are consistent. If they are inconsistent, an annotation error may have occurred.

[0061] Bounding box overlap verification is used to calculate the intersection-over-union (IoU) ratio of bounding boxes for the same detected target in two adjacent frames. If the IoU is too low, such as below a predetermined IoU threshold, but feature matching confirms that it is the same detected target, it may be that the bounding boxes in a certain frame are misaligned.

[0062] Segmentation mask similarity check is used to calculate the similarity of segmentation masks for the same detected target in two adjacent frames. An abnormal similarity may indicate inaccurate segmentation labeling.

[0063] Pose change rationality verification is used to estimate whether the pose change of the detected target between adjacent frames is within a reasonable range, based on the camera's frame rate and motion speed. If the pose change exceeds this reasonable range, there may be an error in the pose annotation.

[0064] In S204, the evaluation result of the algorithm to be evaluated is determined based on the detection result and the standard annotation information.

[0065] The evaluation results refer to the quantitative and qualitative assessments of the performance of the algorithm being evaluated. They include analytical conclusions such as detection accuracy, segmentation accuracy, pose estimation accuracy, and computational performance metrics.

[0066] The process of determining the test results can be as follows: Figure 4 As shown, it includes: In S401, the task fragments are assigned to the computing nodes for processing to obtain the first bounding box region output by the algorithm to be evaluated. Combined with the second bounding box region in the standard annotation information, the intersection-union ratio (IU) used to represent the detection result of the bounding box is determined.

[0067] The first bounding box region is the pixel area covered by the bounding box of the detected target in the image when the algorithm under evaluation processes a certain task segment. The second bounding box region is the pixel area covered by the standard bounding box in the standard annotation information for the same image and the same detected target. The intersection-union ratio (IoU) is a core indicator for measuring the degree of overlap and matching accuracy of two bounding box regions. It represents the ratio of the intersection area to the union area of ​​the two bounding box regions, and the calculation formula can be expressed as: IoU=(A∩B) / (A∪B).

[0068] in, This represents the pixel region corresponding to the first bounding box output by the algorithm under evaluation, i.e., the first bounding box region. This represents the pixel region corresponding to the bounding box in the standard annotation information, i.e., the second bounding box region. express and The intersecting pixel region express and Union of pixel regions; intersection-union ratio The value range is [0,1], and the closer the value is to 1, the higher the detection accuracy.

[0069] In S402, the task fragments are allocated to the computing nodes for processing to obtain the first pixel set corresponding to the segmentation mask output by the algorithm to be evaluated. Combined with the second pixel set corresponding to the segmentation mask in the standard annotation information, a similarity coefficient is determined to represent the detection result of the segmentation mask.

[0070] The first pixel set (X) consists of all pixels with a value of 1 in the segmentation mask output by the algorithm to be evaluated. The second pixel set (Y) is the "ground value" set consisting of all pixels with a value of 1 in the segmentation mask from the standard annotation information.

[0071] The similarity coefficient between the first and second pixel sets can be represented by the Dice similarity coefficient. The Dice similarity coefficient (DSC) can be expressed as: DSC = 2 * (x ∩ Y) / (|X| + |Y|). Here, X represents the pixel set corresponding to the segmentation mask output by the algorithm being evaluated, Y represents the pixel set corresponding to the segmentation mask in the standard annotation information, X ∩ Y represents the number of pixels at the intersection of X and Y, |X| represents the total number of pixels in X, and |Y| represents the total number of pixels in Y. The DSC value ranges from [0, 1], with a value closer to 1 indicating higher segmentation accuracy.

[0072] In S403, the task fragments are assigned to the computing nodes for processing to obtain the first translation vector and the first rotation vector output by the algorithm to be evaluated. Combined with the second translation vector and the second rotation vector in the standard annotation information, a pose estimation index is determined to represent the detection result of pose estimation.

[0073] The first translation vector and the first rotation matrix are the position and orientation of the detected target in the camera coordinate system (or world coordinate system) estimated by the algorithm under evaluation. The second translation vector and the second rotation matrix are the true position and orientation of the detected target provided in the standard annotation information. The pose estimation metric integrates the quantitative indicators of translation error and rotation error and is used to evaluate the overall accuracy of pose estimation.

[0074] Pose error (PE) can be expressed as: .

[0075] in, This represents the translation vector output by the algorithm being evaluated. This represents the translation vector in the standard annotation information. The L2 norm of a vector; This represents the rotation matrix output by the algorithm being evaluated. This represents the rotation matrix in the standard annotation information. Represents the trace of a matrix; The weighting coefficient has a value range of [0,1] and is used to adjust the proportion of translation error and rotation error in the total error. It is configured according to the actual evaluation requirements. The PE value ranges from [0,1], and the closer the value is to 0, the higher the accuracy of pose estimation.

[0076] In S404, the task fragments are allocated to the computing nodes for processing to obtain the computational performance index of the detection results output by the algorithm to be evaluated.

[0077] The computing performance metrics include at least one of throughput, average processing latency, and stability metrics.

[0078] Throughput represents the amount of image data an algorithm can process per unit of time, and is calculated using the following formula:

[0079] in, This indicates the total number of images in a single evaluation. This indicates the number of threads or processes used for parallel processing. Indicate processing Total time taken to process images, from the time the algorithm receives the first image to the time it outputs the last image; throughput. The unit is "sheets / second", and the larger the value, the stronger the large-scale data processing capability.

[0080] Average processing latency represents the average time taken from input to output for a single image. It is calculated as the arithmetic mean of the processing time for a single image.

[0081] The stability index refers to the standard deviation of the average processing delay calculated over multiple evaluations. The smaller the standard deviation, the higher the stability of the algorithm.

[0082] After generating the evaluation results, this application embodiment can also analyze and visualize the evaluation results, perform statistical calculations on various evaluation indicators to obtain data such as mean, standard deviation, and ranking, and perform algorithm accuracy trade-off analysis, scenario adaptability analysis and bottleneck location based on statistical data to determine the matching relationship between different algorithms and scenarios, and generate visual charts to intuitively display the evaluation results.

[0083] In this embodiment, an evaluation report can be automatically generated based on the evaluation results, statistical data, and comprehensive analysis conclusions, and the visualization method or report structure can be adjusted as needed. Furthermore, to improve evaluation performance, after the evaluation task is completed, the system can automatically reset to its initial state, release computing node resources, shut down the algorithm's running process, and archive and store evaluation data, algorithm output results, evaluation index data, and evaluation reports, among other related files. Archived data can be categorized and managed according to the evaluation task name and timestamp for easy subsequent retrieval.

[0084] Taking the humanoid robot in a warehouse setting to detect and identify boxes as an example, the automated evaluation process can be as follows: Figure 5 As shown, it includes: In S501, system initialization and parameter configuration are performed.

[0085] The preprocessing resolution can be set to 1920×1080 and the pose error weighting coefficient can be set through the system configuration interface. =0.5, data shard size is 1000 images / shard, the number of parallel computing nodes is 4 GPU servers, and the Intersection over Union (IoU), similarity coefficient (DSC), pose error (PE), and average latency are selected as the core evaluation metrics. "Evaluator" role permissions are assigned to the operating users, and system configuration modification permissions are restricted.

[0086] In S502, data acquisition and preprocessing are performed.

[0087] It integrates real-world data collected from warehousing scenarios, such as 50,000 box images covering different lighting conditions (natural light, artificial light), occlusion (partial occlusion, complete occlusion), and box sizes (small, medium, large). The annotation information includes bounding boxes, segmentation masks, and pose parameters.

[0088] During the preprocessing process, 120 damaged images and 80 incompletely labeled images can be removed through data verification. The remaining 49,800 images are preprocessed by size normalization, pixel value standardization and Gaussian noise suppression. The labeling information is then converted into standard labeling information in the system standard format and stored.

[0089] In S503, the algorithm to be evaluated is accessed and deployed.

[0090] This embodiment integrates three box recognition algorithms (Algorithm A, Algorithm B, and Algorithm C) to be evaluated. Adaptation is completed through the RPC (Remote Procedure Call) interface. The algorithm version and manufacturer information are uploaded, one A100 GPU resource is allocated to each algorithm, and the Python 3.8 runtime environment and related dependent libraries are configured. Connectivity tests show that all algorithms can receive input and output results normally.

[0091] In S504, large-scale data scheduling and evaluation tasks are executed.

[0092] The remaining standard annotation data can be divided into a predetermined number of task slices, and then the task slices can be allocated to computing nodes. For example, the remaining 49,800 images can be divided into 50 task slices (e.g., the first 49 slices each contain 1,000 images, and the last slice contains 800 images). Through task scheduling, the task slices can be allocated to 4 GPU servers to start parallel evaluation, with each server processing 12-13 slices simultaneously.

[0093] After the detection algorithm completes its processing, the standardized results can be parsed and extracted. The following metrics are calculated: Algorithm A has an average IoU of 0.89, an average DSC of 0.87, an average PE of 0.12, and an average latency of 0.15 seconds; Algorithm B has an average IoU of 0.85, an average DSC of 0.83, an average PE of 0.15, and an average latency of 0.10 seconds; Algorithm C has an average IoU of 0.91, an average DSC of 0.89, an average PE of 0.10, and an average latency of 0.20 seconds.

[0094] In S505, evaluation results are analyzed and visualized.

[0095] The metrics statistics can generate rankings for the three algorithms: IoU and DSC rankings are: Algorithm C > Algorithm A > Algorithm B; PE rankings are: Algorithm C < Algorithm A < Algorithm B; and average latency rankings are: Algorithm B < Algorithm A < Algorithm C.

[0096] The comprehensive analysis revealed that Algorithm C had the best accuracy but the highest latency, Algorithm B had the best performance but slightly lower accuracy, and Algorithm A showed a balanced performance between the two. In occluded scenarios, the IoU of all three algorithms decreased by 10%-15%, which was the main performance bottleneck. Visualization can generate a bar chart comparing algorithm metrics, an IoU trend chart for occluded scenarios, and a comparison chart of Algorithm C and the labeled segmentation mask.

[0097] In S506, evaluation reports are generated and exported.

[0098] It can automatically generate evaluation reports that include an evaluation overview, detailed indicator data, comprehensive analysis conclusions, and suggestions such as "Algorithm C can optimize data processing efficiency, and Algorithm B can improve the detection accuracy of occluded scenes," and export them to a specific format, such as PDF.

[0099] In S507, system reset and data archiving are performed.

[0100] It can free up GPU server resources, shut down algorithm processes, and archive evaluation data, algorithm results, indicator data, and reports to a specified directory on the server, categorized by name as "Warehouse Scenario-20250315".

[0101] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0102] Figure 6 This is a schematic diagram of an automated evaluation device provided in an embodiment of this application. The device includes: The data acquisition unit 601 is used to acquire image data including the target to be detected and the annotation information of the image data, and to deploy the algorithm to be evaluated to multiple computing nodes.

[0103] The standardization processing unit 602 is used to standardize the image data to generate standard image data, and to standardize the annotation information to generate standard annotation information.

[0104] The segmentation processing unit 603 is used to divide the standard image data into multiple task segments and assign the task segments to the computing nodes for processing to obtain detection results. The detection results include at least two of the following: bounding box detection results, segmentation mask, pose estimation results, and runtime.

[0105] The evaluation result determination unit 604 is used to determine the evaluation result of the algorithm to be evaluated based on the detection result and the standard labeling information.

[0106] Figure 6 The automated evaluation device shown is, and Figure 2 The automated evaluation method shown corresponds to this.

[0107] Figure 7 This is a schematic diagram of an automated evaluation system provided in an embodiment of this application. Figure 7As shown, the automated evaluation system 7 of this embodiment includes: a processor 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the processor 70, such as an automated evaluation program. When the processor 70 executes the computer program 72, it implements the steps in the various automated evaluation method embodiments described above. Alternatively, when the processor 70 executes the computer program 72, it implements the functions of each module / unit in the various device embodiments described above.

[0108] For example, the computer program 72 may be divided into one or more modules / units, which are stored in the memory 71 and executed by the processor 70 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 72 in the automated evaluation system 7.

[0109] The automated evaluation system may include, but is not limited to, a processor 70 and a memory 71. Those skilled in the art will understand that... Figure 7 This is merely an example of an automated evaluation system 7 and does not constitute a limitation on the automated evaluation system 7. It may include more or fewer components than shown, or combine certain components, or different components. For example, the automated evaluation system may also include input / output devices, network access devices, buses, etc.

[0110] The processor 70 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0111] The memory 71 can be an internal storage unit of the automated evaluation system 7, such as a hard disk or memory of the automated evaluation system 7. The memory 71 can also be an external storage device of the automated evaluation system 7, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the automated evaluation system 7. Furthermore, the memory 71 can include both internal storage units and external storage devices of the automated evaluation system 7. The memory 71 is used to store the computer program and other programs and data required by the automated evaluation system. The memory 71 can also be used to temporarily store data that has been output or will be output.

[0112] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0113] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0114] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0115] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0116] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0117] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0118] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by hardware related to computer program instructions. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0119] In addition, this application also provides a computer program product that, when run on a computer, causes the computer to execute the methods in the above-described implementations.

[0120] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. An automated evaluation method, characterized in that, The method includes: Acquire image data including the detected target and the annotation information of the image data, and deploy the algorithm to be evaluated to multiple computing nodes; The image data is standardized to generate standard image data, and the annotation information is standardized to generate standard annotation information. The standard image data is divided into multiple task slices, and the task slices are assigned to the computing nodes for processing to obtain detection results. The detection results include at least two of the following: bounding box detection results, segmentation mask, pose estimation results, and runtime. The evaluation result of the algorithm to be evaluated is determined based on the detection results and the standard labeling information.

2. The method according to claim 1, characterized in that, The task is divided and distributed to the computing nodes for processing to obtain detection results, including at least two of the following: The task is divided into segments and distributed to the computing nodes for processing to obtain the first bounding box region output by the algorithm to be evaluated. Combined with the second bounding box region in the standard annotation information, the intersection-union ratio (IU) used to represent the detection result of the bounding box is determined. The task fragments are allocated to the computing nodes for processing to obtain the first set of pixels corresponding to the segmentation mask output by the algorithm to be evaluated. Combined with the second set of pixels corresponding to the segmentation mask in the standard annotation information, a similarity coefficient is determined to represent the detection result of the segmentation mask. The task is divided and distributed to the computing node for processing to obtain the first translation vector and the first rotation vector output by the algorithm to be evaluated. Combined with the second translation vector and the second rotation vector in the standard annotation information, a pose estimation index is determined to represent the detection result of pose estimation. The task is partitioned and distributed to the computing nodes for processing to obtain the computational performance index of the detection result output by the algorithm to be evaluated. The computational performance index includes at least one of throughput, average processing latency and stability index.

3. The method according to claim 1, characterized in that, Before dividing the standard image data into multiple task slices, the method further includes at least one of the following: The standard annotation information is stored in structured data; Consistency verification of standard annotation information for the same detection target is performed through feature matching and geometric verification.

4. The method according to claim 3, characterized in that, The standard annotation information is stored in structured data, including: The structured data stores the correspondence between image ID, target category, bounding box coordinates, segmentation mask, pose parameters, annotation time, and annotation personnel in the standard annotation information; Consistency verification of standard annotation information for the same detection target is performed through feature matching and geometric verification, including: Through feature matching and geometric verification, the consistency of standard annotation information of the same detection target in multiple frames of images is verified, as well as the overlap of bounding boxes, similarity of segmentation bounding boxes, and rationality of pose changes of the same detection target in multiple adjacent frames of images are verified.

5. The method according to claim 1, characterized in that, The standard image data is divided into multiple task slices, including: Determine the feature type of the standard image data; Based on the number of computing nodes, the standard image data with the same feature type is allocated to each computing node, and the task fragments corresponding to each computing node are obtained based on the standard image data allocated to each computing node.

6. The method according to claim 1, characterized in that, Deploy the algorithm to be evaluated to multiple computing nodes, including: The algorithm to be evaluated is received through a standardized interface, and the deployment data of the algorithm to be evaluated is received, including resource configuration information and software environment information. Based on the resource configuration information and software environment information, the algorithm to be evaluated is deployed on at least one computing node.

7. The method according to any one of claims 1-6, characterized in that, The image data is standardized to generate standard image data, including at least one of the following: The image dimensions in the image data are normalized to a predetermined resolution. Map the pixel values ​​in the image data to a preset pixel range; Filter out salt-and-pepper noise and Gaussian noise from the image data; The annotation information is standardized to generate standard annotation information, including at least one of the following: Convert the bounding box detection results into pixel coordinates; Convert the segmentation mask into a binary image format; The pose estimation result is represented as a translation vector and a rotation matrix.

8. An automated evaluation device, characterized in that, The device includes: The data acquisition unit is used to acquire image data including the detected target and the annotation information of the image data, and to deploy the algorithm to be evaluated to multiple computing nodes; The standardization processing unit is used to standardize the image data to generate standard image data, and to standardize the annotation information to generate standard annotation information. The segmentation processing unit is used to divide the standard image data into multiple task segments and assign the task segments to the computing nodes for processing to obtain detection results. The detection results include at least two of the following: bounding box detection results, segmentation mask, pose estimation results, and runtime. The evaluation result determination unit is used to determine the evaluation result of the algorithm to be evaluated based on the detection result and the standard labeling information.

9. An automated evaluation system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it causes the automated evaluation system to implement the method as described in any one of claims 1-7.

10. A computer program product comprising computer program instructions, characterized in that, When the computer program is run, the method as described in any one of claims 1-7 is performed.

Images