A method and related equipment for full-dimensional defect detection of humanoid robots

Abstract

This invention discloses a method and related equipment for full-dimensional defect detection of a humanoid robot. The method acquires robot posture data and performs full-body joint coordinated control of the humanoid robot based on the posture data to obtain thermal imaging data, image data, and 3D point cloud data of the target device. Abnormal regions are identified in the thermal imaging data to obtain intermediate defect regions. Based on the intermediate defect regions, features are extracted from the image data and 3D point cloud data to obtain image features of the image data and point cloud features of the 3D point cloud data. Single-dimensional feature fusion is performed on the point cloud features based on the image features to obtain a single-dimensional fusion result. Based on the robot posture data, the single-dimensional fusion result is updated in all dimensions to obtain the full-dimensional defect detection result of the target device. This method can improve the accuracy and robustness of defect detection in industrial equipment. This invention relates to the field of defect detection technology.

Description

Technical Field

[0001] This invention relates to the field of defect detection technology, and in particular to a method and related equipment for full-dimensional defect detection of humanoid robots. Background Technology

[0002] With the continuous development of society, defect detection of industrial equipment has become a core part of ensuring production safety in industrial settings.

[0003] Currently, in industrial settings, the relevant technologies typically employ traditional robots such as hybrid mobile robots (wheeled / tracked + robotic arms) or quadruped robots to detect defects in industrial equipment. However, since both hybrid mobile robots and quadruped robots lack human-like capabilities for full-body coordination, they have limitations in detection within complex industrial scenarios, resulting in unsatisfactory accuracy and robustness in defect detection.

[0004] Therefore, the problems with the relevant technologies still need to be solved and optimized. Summary of the Invention

[0005] The purpose of this invention is to at least partially solve one of the technical problems existing in the related art.

[0006] Therefore, one objective of this invention is to provide a method and related equipment for full-dimensional defect detection of humanoid robots, wherein the method can improve the accuracy and robustness of defect detection of industrial equipment.

[0007] To achieve the above-mentioned technical objectives, the technical solutions adopted in the embodiments of this application include:

[0008] In a first aspect, embodiments of this application provide a method for full-dimensional defect detection of a humanoid robot, including:

[0009] The robot posture data is acquired, and the humanoid robot is subjected to full-body joint coordinated control based on the robot posture data to obtain multi-source sensor data of the target device, including thermal imaging data, image data and three-dimensional point cloud data.

[0010] The thermal imaging data is used to identify abnormal regions to obtain the intermediate defect region;

[0011] Based on the intermediate defect region, feature extraction is performed on the image data and the three-dimensional point cloud data to obtain the image features of the image data and the point cloud features of the three-dimensional point cloud data.

[0012] Based on the image features, the point cloud features are fused in a single dimension to obtain a single-dimensional fusion result, which corresponds to robot posture data in one detection dimension.

[0013] Based on the robot posture data, the single-dimensional fusion result is updated in all dimensions to obtain the full-dimensional defect detection result of the target device. The full-dimensional defect detection result is fused with the single-dimensional fusion result of robot posture data under several different detection dimensions.

[0014] In addition, the method according to the above embodiments of this application may also have the following additional technical features:

[0015] Furthermore, in one embodiment of this application, the step of identifying abnormal regions from the thermal imaging data to obtain intermediate defect regions includes:

[0016] Obtain the temperature range and area threshold;

[0017] Temperature matrix analysis is performed on the thermal imaging data to obtain the temperature gradient matrix;

[0018] Based on the temperature range, the area threshold, and the temperature gradient matrix, the thermal imaging data is filtered to obtain the intermediate defect region.

[0019] Further, in one embodiment of this application, the step of extracting features from the image data and the 3D point cloud data based on the intermediate defect region to obtain the image features of the image data and the point cloud features of the 3D point cloud data includes:

[0020] Based on the intermediate defect region, the image data is mapped to a first spatial region to obtain a target image region, and based on the intermediate defect region, the three-dimensional point cloud data is mapped to a second spatial region to obtain a target point cloud region.

[0021] Edge and color features are extracted from the target image region to obtain the image features;

[0022] Curvature and depth features are extracted from the target point cloud region to obtain the point cloud features.

[0023] Furthermore, in one embodiment of this application, the step of performing single-dimensional feature fusion on the point cloud features based on the image features to obtain a single-dimensional fusion result includes:

[0024] Obtain feature weight reassemblies corresponding to the image features and the point cloud features, wherein the feature weight reassemblies include several dynamic weights;

[0025] Based on the feature weight reorganization, the image features and the point cloud features are weighted and fused to obtain the single-dimensional fusion result.

[0026] Furthermore, in one embodiment of this application, the feature weight reorganization is obtained through the following steps:

[0027] Obtain the rule weight table and the current environmental data of the target device. The rule weight table records several original weight sets and the rule parameters corresponding to each original weight set.

[0028] Based on the current environment data and the robot posture data, rule matching is performed on all the rule parameters to obtain the target parameters;

[0029] Based on the target parameters, all the original weight recombinations are mapped and filtered to obtain the feature weight recombinations.

[0030] Furthermore, in one embodiment of this application, the step of updating the single-dimensional fusion result in all dimensions based on the robot posture data to obtain the full-dimensional defect detection result of the target device includes:

[0031] Dimension detection is performed on the current single-dimensional fusion result to obtain a dimension detection result, which is used to indicate whether the current single-dimensional fusion result is the single-dimensional fusion result of the last detected dimension;

[0032] If the dimension detection result is that the current single-dimensional fusion result is not the single-dimensional fusion result of the last detected dimension, then the current single-dimensional fusion result is retained and the robot posture data is updated. Then, the process returns to obtain the robot posture data and perform full-body joint coordinated control of the humanoid robot based on the robot posture data. Alternatively, if the dimension detection result is that the current single-dimensional fusion result is the single-dimensional fusion result of the last detected dimension, then all single-dimensional fusion results are fused in all dimensions to obtain the full-dimensional defect detection result.

[0033] Furthermore, in one embodiment of this application, the step of performing full-dimensional fusion on all the single-dimensional fusion results to obtain the full-dimensional defect detection result includes:

[0034] Obtain the confidence threshold and the dimension weight corresponding to each of the detection dimensions;

[0035] Based on all the dimension weights, the corresponding single-dimensional fusion results are complementaryly fused to obtain the full-dimensional fusion result;

[0036] Based on the confidence threshold, a confidence analysis is performed on the full-dimensional fusion result to obtain the full-dimensional defect detection result.

[0037] Secondly, embodiments of this application provide a full-dimensional defect detection system for a humanoid robot, comprising:

[0038] The first processing unit is used to acquire robot posture data and perform full-body joint coordinated control of the humanoid robot based on the robot posture data to obtain multi-source sensor data of the target device. The multi-source sensor data includes thermal imaging data, image data and three-dimensional point cloud data.

[0039] The second processing unit is used to identify abnormal regions in the thermal imaging data to obtain the intermediate defect region.

[0040] The third processing unit is used to extract features from the image data and the three-dimensional point cloud data based on the intermediate defect region, so as to obtain the image features of the image data and the point cloud features of the three-dimensional point cloud data.

[0041] The fourth processing unit is used to perform single-dimensional feature fusion on the point cloud features based on the image features to obtain a single-dimensional fusion result, wherein the single-dimensional fusion result corresponds to robot posture data in one detection dimension.

[0042] The fifth processing unit is used to update the single-dimensional fusion result in all dimensions based on the robot posture data to obtain the full-dimensional defect detection result of the target device. The full-dimensional defect detection result is fused with the single-dimensional fusion result of robot posture data under several different detection dimensions.

[0043] Thirdly, embodiments of this application also provide an electronic device, including:

[0044] At least one processor;

[0045] At least one memory for storing at least one program;

[0046] When the at least one program is executed by the at least one processor, the at least one processor performs the method described above.

[0047] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a processor-executable program, which, when executed by the processor, is used to implement the above-described method.

[0048] The advantages and beneficial effects of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application:

[0049] This application discloses a method and related equipment for full-dimensional defect detection of a humanoid robot. The method acquires robot posture data and performs full-body joint coordinated control of the humanoid robot based on the robot posture data to obtain multi-source sensor data of the target device. The multi-source sensor data includes thermal imaging data, image data, and 3D point cloud data. Abnormal regions are identified in the thermal imaging data to obtain intermediate defect regions. Based on the intermediate defect regions, features are extracted from the image data and 3D point cloud data to obtain image features of the image data and point cloud features of the 3D point cloud data. Based on the image features, single-dimensional feature fusion is performed on the point cloud features to obtain a single-dimensional fusion result, which corresponds to robot posture data in one detection dimension. Based on the robot posture data, the single-dimensional fusion result is updated across all dimensions to obtain a full-dimensional defect detection result for the target device. The full-dimensional defect detection result incorporates single-dimensional fusion results from several different detection dimensions of robot posture data. This method obtains single-dimensional fusion results for each detection dimension using a humanoid robot, which can overcome the detection blind spots and posture limitations of traditional robots in complex industrial scenarios. Furthermore, by updating the single-dimensional fusion results across all dimensions, this method can fuse fusion results from different detection dimensions, effectively improving the accuracy and robustness of defect detection. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of this application or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions in this application. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0051] Figure 1 A flowchart illustrating a method for full-dimensional defect detection of a humanoid robot provided in an embodiment of this application;

[0052] Figure 2 A schematic diagram of the framework of a full-dimensional defect detection system for a humanoid robot provided in an embodiment of this application;

[0053] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0054] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0056] Currently, in industrial settings, the relevant technologies typically employ hybrid mobile robots (wheeled / tracked + robotic arm) or quadruped robots, or other traditional robots, to detect defects in industrial equipment. Specifically, hybrid mobile robots rely on wheeled or tracked chassis for movement, while the robotic arm handles the inspection operations. However, in complex industrial scenarios, the following issues arise:

[0057] 1) Limited terrain adaptability: It can only move efficiently on flat ground. It is easy to get stuck when facing obstacles such as workshop steps (>10cm), gaps and grooves between equipment, etc., requiring an additional lifting mechanism (increasing complexity by more than 30%).

[0058] 2) Limited detection angle: The robotic arm has limited degrees of freedom (6-8 degrees of freedom) and is installed on a fixed chassis, making it difficult to achieve full-angle detection around complex equipment structures (such as the inside of pipe bends and tank heads), resulting in 20%-30% blind spots in detection.

[0059] 3) Insufficient vertical coverage: It relies on the extension of the robotic arm to compensate for the height. When detecting at high altitudes, the data jitters due to the shift in the center of gravity (point cloud error > 5mm). When detecting at low altitudes, the robotic arm needs to be bent excessively (which is prone to collision with the ground).

[0060] Quadruped robots improve terrain stability through four-legged support and combine this with robotic arms, but their detection capabilities are still limited by their structure, as shown in the following situations:

[0061] 1) Limited vertical space coverage: The height of the machine body is fixed (0.5-1m). Even if the maximum detection height of the robot arm is ≤2m, it cannot cover equipment areas above 2m (such as pipe supports and the top of storage tanks).

[0062] 2) Insufficient redundancy in operating posture: The robotic arm is mounted on the back or shoulder, and the posture detection depends on the overall movement of the body (the time for adjusting the four-legged position increases by 50%), making it difficult to achieve human-like fine-tuning movements such as "bending over" and "extending to the side".

[0063] 3) Poor human-machine collaboration compatibility: Animal-like movement patterns (quadruped gait) lack predictability, requiring safety isolation (limiting the detection range) in human-machine hybrid scenarios, and cannot be integrated into human workflows.

[0064] Furthermore, since both hybrid mobile robots and quadruped robots lack human-like capabilities for full-body coordination, they have limitations in detection within complex industrial scenarios, specifically in the following situations:

[0065] 1) Multimodal data (thermal imaging, images, point clouds) exhibit spatiotemporal mismatch due to attitude instability (error > 10ms);

[0066] 2) Under environmental interference (low light, high reflectivity), the feature weights cannot be dynamically adapted, and the recognition accuracy drops to below 60%;

[0067] 3) It cannot respond to human collaborative commands (such as gestures to guide the detection area), resulting in high deployment costs.

[0068] The aforementioned issues with quadruped robots and / or hybrid mobile robots result in unsatisfactory accuracy and robustness in defect detection.

[0069] It should be noted that the aforementioned related technologies are only used to assist in understanding the technical solutions of this application and do not mean that they belong to the publicly disclosed prior art.

[0070] In view of this, this application provides a method and related equipment for full-dimensional defect detection of humanoid robots. The method obtains single-dimensional fusion results for each detection dimension through the humanoid robot. Specifically, it is based on the unique advantage of humanoid robots, which have a humanoid structure with bipedal movement and coordinated control of all joints, to better adapt to the complex needs of industrial scenarios and effectively solve the problems of detection blind spots, data inaccuracy, and poor scene adaptation of traditional robots, thereby effectively improving the accuracy and robustness of defect detection.

[0071] Furthermore, this method specifically performs weighted fusion of image features and point cloud features through dynamic weights. This allows for local adaptation of the weight relationships between image features and point cloud features within a single dimension. Additionally, it uses dimensional weights to perform complementary fusion of all single-dimensional fusion results, enabling global adaptation of weight relationships between different dimensions. This dynamic weight adaptation from both local and global perspectives helps improve the accuracy and robustness of subsequent defect identification.

[0072] Reference Figure 1In this embodiment of the application, a method for full-dimensional defect detection of a humanoid robot includes:

[0073] Step 110: Obtain robot posture data, and perform full-body joint coordinated control of the humanoid robot based on the robot posture data to obtain multi-source sensor data of the target device. The multi-source sensor data includes thermal imaging data, image data, and three-dimensional point cloud data.

[0074] In this embodiment, for any single-dimensional defect detection process in the full-dimensional defect detection of a humanoid robot, robot posture data is used to instruct the humanoid robot on the posture required for the current single-dimensional defect detection. Specifically, if the current single-dimensional defect detection process is the first single-dimensional defect detection process in all single-dimensional defect detection processes, the robot posture data can be preset posture data, and the multi-source sensor data collected by the humanoid robot in this robot posture data is recorded as the sensor data of the initial viewpoint (0°); or, if the current single-dimensional defect detection process is the second or subsequent single-dimensional defect detection process in all single-dimensional defect detection processes, the robot posture data can be the updated robot posture data in the previous single-dimensional defect detection process, and the multi-source sensor data collected by the humanoid robot in this robot posture data can be sensor data of the side viewpoint (-30°), pitch / oblique viewpoint (+30°), etc.; the viewpoint corresponding to the multi-source sensor data in each single-dimensional defect detection process is different.

[0075] It is understandable that for any single-dimensional defect detection process, multi-source sensor data of the target device can be collected by multi-source sensors mounted on the humanoid robot, based on the coordinated control of the humanoid robot's whole-body joints (such as adjusting the lower limbs' standing position, rotating the torso, and adjusting the upper limbs' posture). For example, thermal imaging data can be obtained through the thermal imaging module in the multi-source sensors, image data can be obtained through the visual acquisition module in the multi-source sensors, and three-dimensional point cloud data can be obtained through the lidar module in the multi-source sensors. The target device can be the device to be detected in an industrial scenario. There are already many types of specific devices, which will not be elaborated here.

[0076] It should be noted that adjusting the humanoid robot's lower limbs can be done by adjusting the hip and knee joints to ensure the detection distance is within a preset range. Torso rotation can be achieved by rotating the waist ±90° to adjust orientation, and by adjusting the waist angle (e.g., -30° to +30°) to accommodate different heights. Upper limb posture adjustment can be achieved by adjusting the shoulder joint ±180°, the elbow joint 0° to 150°, and the wrist, among other things.

[0077] Step 120: Identify abnormal regions in the thermal imaging data to obtain the intermediate defect region;

[0078] In this embodiment of the application, abnormal area identification can be defined as an intermediate defect area in the thermal imaging data where the surface temperature of the target device is abnormal.

[0079] In some embodiments, the step of identifying abnormal regions from the thermal imaging data to obtain intermediate defect regions includes:

[0080] Obtain the temperature range and area threshold;

[0081] Temperature matrix analysis is performed on the thermal imaging data to obtain the temperature gradient matrix;

[0082] Based on the temperature range, the area threshold, and the temperature gradient matrix, the thermal imaging data is filtered to obtain the intermediate defect region.

[0083] In this embodiment, abnormal area identification can first involve obtaining the temperature range of the target device under normal operating conditions [T0-ΔT, T0+ΔT], and the maximum area of ​​a preset temperature abnormal area, denoted as the area threshold. This area threshold can be, for example, 5cm. 2 .

[0084] It is understandable that temperature matrix analysis can construct a temperature gradient matrix from various temperature data in thermal imaging data. There are already various ways to implement this temperature gradient matrix, which will not be elaborated upon here. Range filtering can be based on temperature range, area threshold, and the temperature value of each matrix element in the temperature gradient matrix to filter and extract temperature anomaly regions in the thermal imaging data. Specifically, regions with temperature values ​​exceeding the temperature range [T0-ΔT, T0+ΔT] and an area threshold ≥ 5cm² can be selected. 2 The area identified from the thermal imaging data was determined to be the intermediate defect region.

[0085] Step 130: Based on the intermediate defect region, perform feature extraction on the image data and the three-dimensional point cloud data to obtain the image features of the image data and the point cloud features of the three-dimensional point cloud data;

[0086] In this embodiment, feature extraction can be based on the target device region indicated by the intermediate defect region, extracting features of the target device region corresponding to the image data and the three-dimensional point cloud data respectively, thereby obtaining image features and point cloud features.

[0087] In some embodiments, the step of extracting features from the image data and the 3D point cloud data based on the intermediate defect region to obtain image features of the image data and point cloud features of the 3D point cloud data includes:

[0088] Based on the intermediate defect region, the image data is mapped to a first spatial region to obtain a target image region, and based on the intermediate defect region, the three-dimensional point cloud data is mapped to a second spatial region to obtain a target point cloud region.

[0089] Edge and color features are extracted from the target image region to obtain the image features;

[0090] Curvature and depth features are extracted from the target point cloud region to obtain the point cloud features.

[0091] In this embodiment of the application, the first spatial region mapping can be determined by using a spatial coordinate mapping algorithm to determine the target image region in the image data corresponding to the intermediate defect region; while the second spatial region mapping can be determined by using a spatial coordinate mapping algorithm to determine the target point cloud region in the three-dimensional point cloud data corresponding to the intermediate defect region.

[0092] It is understandable that edge and color feature extraction can be performed by calculating the edge feature values ​​of the target image region using the Canny operator, and extracting the color feature values ​​of the target image region based on the HSV space, and then determining the obtained edge and color feature values ​​as image features. Curvature and depth feature extraction can be performed by extracting the curvature feature values ​​of the target point cloud region through covariance matrix decomposition, and by calculating the distance deviation between each point cloud within the target point cloud region and the reference plane to determine the depth feature values, and then determining the obtained curvature and depth feature values ​​as point cloud features.

[0093] Step 140: Based on the image features, perform single-dimensional feature fusion on the point cloud features to obtain a single-dimensional fusion result, wherein the single-dimensional fusion result corresponds to robot posture data in one detection dimension;

[0094] In this embodiment of the application, single-dimensional feature fusion can be the fusion of image features and point cloud features within a dimension to obtain a single-dimensional fusion result. The detection dimension corresponding to the single-dimensional fusion result can be the acquisition perspective of the multi-source sensor data corresponding to the humanoid robot after performing full-body joint coordinated control based on robot posture data.

[0095] Understandably, in practical applications, after obtaining the single-dimensional fusion result, a pre-trained SVM classifier can be used to identify the feature combination in the single-dimensional fusion result to obtain the defect type of the target device in the current detection dimension, and then add the defect type to the single-dimensional fusion result.

[0096] In some embodiments, the step of performing single-dimensional feature fusion on the point cloud features based on the image features to obtain a single-dimensional fusion result includes:

[0097] Obtain feature weight reassemblies corresponding to the image features and the point cloud features, wherein the feature weight reassemblies include several dynamic weights;

[0098] Based on the feature weight reorganization, the image features and the point cloud features are weighted and fused to obtain the single-dimensional fusion result.

[0099] In this embodiment, for the current single-dimensional defect detection process, single-dimensional feature fusion can be achieved by obtaining feature weight reassembly corresponding to image features and point cloud features. The feature weight reassembly includes several dynamic weights, with each image feature and point cloud feature corresponding to a different dynamic weight. Specifically, edge feature values ​​in the image features correspond to one dynamic weight, and color feature values ​​in the image features correspond to one dynamic weight; while curvature feature values ​​in the point cloud features correspond to one dynamic weight, and depth feature values ​​in the point cloud features correspond to one dynamic weight.

[0100] It is understandable that weighted fusion can calculate the fusion matching degree of image features and point cloud features in a single dimension, thereby obtaining a single-dimensional fusion result, which can be expressed as:

[0101]

[0102] in, This is a single-dimensional fusion result; These are the normalized edge feature values ​​in the image features; for The corresponding dynamic weights; These are the normalized color feature values ​​in the image features; for The corresponding dynamic weights; The normalized curvature feature value in the point cloud features; for The corresponding dynamic weights; The normalized depth feature value in the point cloud features; for The corresponding dynamic weights.

[0103] In some embodiments, the feature weight reorganization is obtained through the following steps:

[0104] Obtain the rule weight table and the current environmental data of the target device. The rule weight table records several original weight sets and the rule parameters corresponding to each original weight set.

[0105] Based on the current environment data and the robot posture data, rule matching is performed on all the rule parameters to obtain the target parameters;

[0106] Based on the target parameters, all the original weight recombinations are mapped and filtered to obtain the feature weight recombinations.

[0107] In this embodiment, the current environmental data may be light intensity, device environment, ambient temperature, etc., related to the target device. Rule parameters are used to indicate several conditions, which may specifically be environmental conditions and / or attitude conditions, etc. For example, a rule weight table provided in this embodiment is shown in Table 1 below:

[0108] Table 1

[0109]

[0110] It is understandable that in Table 1 This is a thermal imaging-derived weight, which can be enabled under highly reflective conditions to replace dynamic weights of color feature values ​​in image features. Rule matching can be based on the current environment data and robot posture data to match the conditions in the rule parameters, thereby determining the rule parameters corresponding to the current environment data and robot posture data from the rule weight table, which are denoted as target parameters; then, based on the mapping relationship between each rule parameter in the rule weight table and the original weight reassembly, the original weight reassembly of the target parameters is determined, resulting in the feature weight reassembly.

[0111] It should be noted that, for ease of understanding, the rule weight table provided in this application embodiment is only a simple example and is not intended to limit the rule weight table. In practical applications, there are often multiple superimposed conditions, and the corresponding original weight reassemblies can be flexibly transformed, which will not be elaborated here. Furthermore, if the target rule parameter has exactly one condition, its original weight reassembly can be obtained through pre-setting; or, if the number of conditions in the target rule parameter is greater than or equal to 2, its original weight reassembly can be obtained through the following steps:

[0112] 1) Obtain condition priority information, which specifically includes:

[0113] 1-a. Equipment material (high reflectivity): High reflectivity has the most significant impact on color characteristics. The equipment material is usually a metal tank, polished pipe, etc., which is the first priority.

[0114] 1-b. Illumination intensity (low light / high light): Illumination directly affects image quality, second priority;

[0115] 1-c. Ambient temperature (fluctuation): Temperature fluctuations affect thermal imaging and color features, third priority;

[0116] 1-d, Robot posture (tilt): Posture tilt affects the spatial mapping between point cloud and image, fourth priority.

[0117] 2) Based on the condition priority information, determine the priority of several conditions in the target rule parameters to obtain the priority of each condition in the target rule parameters;

[0118] 3) Load the basic weight reorganization corresponding to the high-priority conditions in the target rule parameters, and make secondary adjustments to the basic weight reorganization through the low-priority conditions to obtain the original weight reorganization.

[0119] For example, this application embodiment takes a "low light + high reflectivity" scenario as an example (i.e., simultaneously satisfying "light intensity"). "and the equipment material is highly reflective" ():

[0120] Priority determination: High reflectivity (material conditions) > Low illumination (illumination conditions);

[0121] Basic weight selection: Prioritize loading basic weight reassembly for high-reflectivity scenes ( );

[0122] Low-light condition correction: Weights (color) that are strongly correlated with illumination in highly reflective scenes. ,edge A second adjustment will be made:

[0123] Because there is no edge weight in highly reflective scenes Supplementing low-light scenes For example ;

[0124] The final reorganization of original rights: .

[0125] It is worth mentioning that after the original weight reassembly obtained in this application, it is also possible to choose whether to constrain the total weight in the original weight reassembly to 1 according to actual needs. For example, the total weight of the original weight group in the single condition (attitude) in Table 1 can be 1. There are already various specific methods for constraining the total weight, such as normalizing the weights of the original weight group again, which will not be elaborated here. In addition, in another implementation, such as in a highly reflective scene, the aforementioned single-dimensional fusion result can be adaptively adjusted as follows:

[0126]

[0127] in, Thermal imaging features of thermal imaging data; for The dynamic weights, also known as thermal imaging derived weights.

[0128] Step 150: Based on the robot posture data, update the single-dimensional fusion result in all dimensions to obtain the full-dimensional defect detection result of the target device. The full-dimensional defect detection result is fused with the single-dimensional fusion result of robot posture data under several different detection dimensions.

[0129] In this embodiment of the application, the full-dimensional update can be carried out in a cyclical manner. After the humanoid robot performs full-body joint coordinated control based on the robot posture data, it obtains the single-dimensional fusion result under each detection dimension, and then fuses the single-dimensional fusion results under all detection dimensions to obtain the full-dimensional defect detection result of the target device.

[0130] In some embodiments, updating the single-dimensional fusion result across all dimensions based on the robot posture data to obtain the full-dimensional defect detection result of the target device includes:

[0131] Dimension detection is performed on the current single-dimensional fusion result to obtain a dimension detection result, which is used to indicate whether the current single-dimensional fusion result is the single-dimensional fusion result of the last detected dimension;

[0132] If the dimension detection result is not the single-dimensional fusion result of the last detected dimension, then the current single-dimensional fusion result is retained and the robot posture data is updated. Then, the process returns to the step of obtaining robot posture data and performing full-body joint coordinated control of the humanoid robot based on the robot posture data.

[0133] In this embodiment, dimension detection can be performed by detecting whether the current single-dimensional fusion result is the single-dimensional fusion result of the last detected dimension in the entire loop process, thereby obtaining the dimension detection result. Specifically, if the dimension detection result is the current single-dimensional fusion result but not the single-dimensional fusion result of the last detected dimension, it indicates that the loop has not ended. At this time, the current single-dimensional fusion result can be retained, and the robot posture data can be updated according to the robot posture parameters required for the next single-dimensional defect detection process, and then the process can return to step 110.

[0134] Alternatively, if the current single-dimensional fusion result is the single-dimensional fusion result of the last detected dimension, then all single-dimensional fusion results are fused in all dimensions to obtain the full-dimensional defect detection result.

[0135] Furthermore, the step of performing full-dimensional fusion on all the single-dimensional fusion results to obtain the full-dimensional defect detection results includes:

[0136] Obtain the confidence threshold and the dimension weight corresponding to each of the detection dimensions;

[0137] Based on all the dimension weights, the corresponding single-dimensional fusion results are complementaryly fused to obtain the full-dimensional fusion result;

[0138] Based on the confidence threshold, a confidence analysis is performed on the full-dimensional fusion result to obtain the full-dimensional defect detection result.

[0139] In this embodiment, the dimensional weights can be pre-set weights or simply derived from the aforementioned content regarding dynamic weights, which will not be elaborated upon here. This embodiment takes a total of 3 single-dimensional fusion results as an example. Complementary fusion can be calculated based on each single-dimensional fusion result and its corresponding dimensional weight to obtain a full-dimensional fusion result, which can be expressed as:

[0140]

[0141] in, For a fully integrated result; This represents the single-dimensional fusion result under the initial viewpoint (0°) detection dimension; This is the single-dimensional fusion result under the detection dimension of the side view (-30°); This is the single-dimensional fusion result under the initial viewing angle (+30°) detection dimension; , and As the parameters are adjustable, the embodiments of this application use... , and For example, the values ​​are 0.5, 0.25, and 0.25 respectively.

[0142] Understandably, in practical applications, the types of defects in target equipment typically include cracks, corrosion, and deformation. The initial view provides the most complete information, while side / tilt views supplement blind spot information. For cracks, the edge features are more prominent in the side view (-30°), which can correct for omissions in the initial view. For corrosion, the color and depth features are clearer in the tilt view, which can verify the extent of corrosion. For deformation, the curvature features from multiple views work together to verify the defects, which can help reduce the false positive rate.

[0143] It should be noted that confidence analysis can compare the specific value of the full-dimensional fusion result with the confidence threshold, and when the specific value of the full-dimensional fusion result is greater than or equal to the confidence threshold, the defect type of the full-dimensional fusion result is determined as the full-dimensional defect detection result.

[0144] The following describes in detail, with reference to the accompanying drawings, a full-dimensional defect detection system for a humanoid robot according to an embodiment of this application.

[0145] Reference Figure 2 The embodiment of this application proposes a full-dimensional defect detection system for a humanoid robot, comprising:

[0146] The first processing unit 101 is used to acquire robot posture data and perform full-body joint coordinated control of the humanoid robot based on the robot posture data to obtain multi-source sensor data of the target device. The multi-source sensor data includes thermal imaging data, image data and three-dimensional point cloud data.

[0147] The second processing unit 102 is used to identify abnormal regions in the thermal imaging data to obtain intermediate defect regions.

[0148] The third processing unit 103 is used to extract features from the image data and the three-dimensional point cloud data based on the intermediate defect region, so as to obtain the image features of the image data and the point cloud features of the three-dimensional point cloud data.

[0149] The fourth processing unit 104 is used to perform single-dimensional feature fusion on the point cloud features according to the image features to obtain a single-dimensional fusion result, wherein the single-dimensional fusion result corresponds to robot posture data of a detection dimension.

[0150] The fifth processing unit 105 is used to update the single-dimensional fusion result in all dimensions according to the robot posture data to obtain the full-dimensional defect detection result of the target device. The full-dimensional defect detection result is fused with the single-dimensional fusion result of robot posture data under several different detection dimensions.

[0151] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0152] Reference Figure 3 This application also provides an electronic device, including:

[0153] At least one processor 201;

[0154] At least one memory 202 is used to store at least one program;

[0155] When the at least one program is executed by the at least one processor 201, the at least one processor 201 implements the above-described method embodiments.

[0156] Similarly, it can be understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0157] This application also provides a computer-readable storage medium storing a program executable by a processor 201, which, when executed by the processor 201, is used to implement the above-described method embodiments.

[0158] Similarly, the content of the above method embodiments is applicable to the present computer-readable storage medium embodiments. The specific functions implemented by the present computer-readable storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0159] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0160] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods.

[0161] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this application are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0162] Furthermore, although this application is described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding this application. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional technology for an engineer. Therefore, those skilled in the art can implement the application set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of this application, which is determined by the full scope of the appended claims and their equivalents.

[0163] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods in the embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0164] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0165] More specific examples (a non-exhaustive list) of computer-readable media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0166] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0167] In the foregoing description of this specification, the references to terms such as "one embodiment," "another embodiment," or "some embodiments," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0168] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

[0169] The above is a detailed description of the preferred embodiments of this application, but this application is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A full-dimension defect detection method of a humanoid robot, characterized by, include: The robot posture data is acquired, and the humanoid robot is subjected to full-body joint coordinated control based on the robot posture data to obtain multi-source sensor data of the target device, including thermal imaging data, image data and three-dimensional point cloud data. The thermal imaging data is used to identify abnormal regions to obtain the intermediate defect region; Based on the intermediate defect region, feature extraction is performed on the image data and the three-dimensional point cloud data to obtain the image features of the image data and the point cloud features of the three-dimensional point cloud data. Based on the image features, the point cloud features are fused in a single dimension to obtain a single-dimensional fusion result. The single-dimensional fusion result corresponds to robot posture data in one detection dimension. The detection dimension is used to characterize the viewpoint dimension when the humanoid robot collects the multi-source sensor data based on the robot posture data. Based on the robot posture data, the single-dimensional fusion result is updated in all dimensions to obtain the full-dimensional defect detection result of the target device. The full-dimensional defect detection result is fused with the single-dimensional fusion result of robot posture data under several different detection dimensions. The step of updating the single-dimensional fusion result in all dimensions based on the robot posture data to obtain the full-dimensional defect detection result of the target device includes: Dimension detection is performed on the current single-dimensional fusion result to obtain a dimension detection result, which is used to indicate whether the current single-dimensional fusion result is the single-dimensional fusion result of the last detected dimension; If the dimension detection result is that the current single-dimensional fusion result is not the single-dimensional fusion result of the last detected dimension, then the current single-dimensional fusion result is retained and the robot posture data is updated. Then, the process returns to obtain the robot posture data and perform full-body joint coordinated control of the humanoid robot based on the robot posture data. Alternatively, if the dimension detection result is that the current single-dimensional fusion result is the single-dimensional fusion result of the last detected dimension, then all single-dimensional fusion results are fused in all dimensions to obtain the full-dimensional defect detection result.

2. The method according to claim 1, characterized in that, The step of identifying abnormal regions from the thermal imaging data to obtain intermediate defect regions includes: Obtain the temperature range and area threshold; Temperature matrix analysis is performed on the thermal imaging data to obtain the temperature gradient matrix; Based on the temperature range, the area threshold, and the temperature gradient matrix, the thermal imaging data is filtered to obtain the intermediate defect region.

3. The method according to claim 1, characterized in that, The step of extracting features from the image data and 3D point cloud data based on the intermediate defect region to obtain image features of the image data and point cloud features of the 3D point cloud data includes: Based on the intermediate defect region, the image data is mapped to a first spatial region to obtain a target image region, and based on the intermediate defect region, the three-dimensional point cloud data is mapped to a second spatial region to obtain a target point cloud region. Edge and color features are extracted from the target image region to obtain the image features; Curvature and depth features are extracted from the target point cloud region to obtain the point cloud features.

4. The method according to claim 1, characterized in that, The step of performing single-dimensional feature fusion on the point cloud features based on the image features to obtain a single-dimensional fusion result includes: Obtain feature weight reassemblies corresponding to the image features and the point cloud features, wherein the feature weight reassemblies include several dynamic weights; Based on the feature weight reorganization, the image features and the point cloud features are weighted and fused to obtain the single-dimensional fusion result.

5. The method according to claim 4, characterized in that, The feature weight reorganization is obtained through the following steps: Obtain the rule weight table and the current environmental data of the target device. The rule weight table records several original weight sets and the rule parameters corresponding to each original weight set. Based on the current environment data and the robot posture data, rule matching is performed on all the rule parameters to obtain the target parameters; Based on the target parameters, all the original weight recombinations are mapped and filtered to obtain the feature weight recombinations.

6. The method according to claim 1, characterized in that, The step of performing full-dimensional fusion on all the single-dimensional fusion results to obtain the full-dimensional defect detection results includes: Obtain the confidence threshold and the dimension weight corresponding to each of the detection dimensions; Based on all the dimension weights, the corresponding single-dimensional fusion results are complementaryly fused to obtain the full-dimensional fusion result; Based on the confidence threshold, a confidence analysis is performed on the full-dimensional fusion result to obtain the full-dimensional defect detection result.

7. A full-dimensional defect detection system for a humanoid robot, characterized in that, include: The first processing unit is used to acquire robot posture data and perform full-body joint coordinated control of the humanoid robot based on the robot posture data to obtain multi-source sensor data of the target device. The multi-source sensor data includes thermal imaging data, image data and three-dimensional point cloud data. The second processing unit is used to identify abnormal regions in the thermal imaging data to obtain the intermediate defect region. The third processing unit is used to extract features from the image data and the three-dimensional point cloud data based on the intermediate defect region, so as to obtain the image features of the image data and the point cloud features of the three-dimensional point cloud data. The fourth processing unit is used to perform single-dimensional feature fusion on the point cloud features according to the image features to obtain a single-dimensional fusion result, wherein the single-dimensional fusion result corresponds to robot posture data of a detection dimension; the detection dimension is used to characterize the view dimension when the humanoid robot collects the multi-source sensor data based on the robot posture data. The fifth processing unit is used to update the single-dimensional fusion result in all dimensions based on the robot posture data to obtain the full-dimensional defect detection result of the target device. The full-dimensional defect detection result is fused with the single-dimensional fusion result of robot posture data under several different detection dimensions. The step of updating the single-dimensional fusion result in all dimensions based on the robot posture data to obtain the full-dimensional defect detection result of the target device includes: Dimension detection is performed on the current single-dimensional fusion result to obtain a dimension detection result, which is used to indicate whether the current single-dimensional fusion result is the single-dimensional fusion result of the last detected dimension; If the dimension detection result is that the current single-dimensional fusion result is not the single-dimensional fusion result of the last detected dimension, then the current single-dimensional fusion result is retained and the robot posture data is updated. Then, the process returns to obtain the robot posture data and perform full-body joint coordinated control of the humanoid robot based on the robot posture data. Alternatively, if the dimension detection result is that the current single-dimensional fusion result is the single-dimensional fusion result of the last detected dimension, then all single-dimensional fusion results are fused in all dimensions to obtain the full-dimensional defect detection result.

8. An electronic device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor performs the method as described in any one of claims 1-6.

9. A computer-readable storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to implement the method as described in any one of claims 1-6.

Images