A laser industrial quality detection method and device based on semantic perception

Abstract

The application provides a laser industrial quality detection method and device based on semantic perception, and the method comprises the following steps: acquiring multi-modal data information of an industrial area to be detected; performing space-time calibration on the multi-modal data information to obtain a multi-dimensional feature vector of each sampling point after fusion; inputting the multi-dimensional feature vector of each sampling point into a pre-trained semantic segmentation network for semantic segmentation to generate a three-dimensional semantic map; acquiring a detection instruction, analyzing and target reasoning the detection instruction to obtain a detection target coordinate; performing path planning according to the detection target coordinate and the three-dimensional semantic map to obtain an optimal execution path for moving to a target position; moving to the target position according to the optimal execution path, and selecting a corresponding scanning mode according to the information of the three-dimensional semantic map to perform quality detection on the target to obtain a corresponding detection result; thereby improving the intelligent level and detection efficiency of the quality detection robot.

Description

Technical Field

[0001] This invention relates to the field of industrial inspection and intelligent manufacturing technology, and in particular to a laser industrial quality inspection method based on semantic perception, a laser industrial quality inspection device based on semantic perception, a computer-readable storage medium, and a computer device. Background Technology

[0002] As industrial manufacturing moves towards intelligence and precision, laser inspection technology, with its advantages of high precision, non-contact operation, and rapid response, is widely used in industrial quality inspection scenarios such as component geometry measurement, surface defect detection, and weld quality analysis. However, existing laser quality inspection systems still have the following shortcomings: First, the inspection task relies on manually setting the inspection path and target coordinates. In complex industrial environments, the positions of parts are constantly changing and the scene is severely occluded. Traditional methods require manual input or pre-calibration of the spatial position of the inspection object, which is difficult to meet the needs of automated inspection for multiple stations and batches, resulting in low inspection efficiency and long deployment cycles.

[0003] Secondly, existing navigation and positioning methods lack semantic understanding capabilities. Traditional laser quality inspection robots mostly rely on two-dimensional maps or geometric modeling for path planning, which cannot identify semantic-level information such as "target equipment," "inspection area," and "dangerous area." As a result, they are prone to path failure, increased collision risk, and insufficient task flexibility when the environment changes.

[0004] Secondly, the quality inspection robot lacks intelligent interaction and decision-making capabilities with the inspection task. In most systems, the inspection task is triggered by fixed instructions, and the robot cannot understand the inspection intention and autonomously locate the target through natural language instructions (such as "inspect the weld area" or "check the left flange thread"). This "instruction-based" interaction method restricts the system's intelligence level and ease of operation. Summary of the Invention

[0005] This invention aims to at least partially solve one of the technical problems in the aforementioned technologies. To this end, one objective of this invention is to propose a semantic-aware laser industrial quality inspection method. By introducing an improved 3D semantic modeling network, object reasoning algorithm, and semantic-aware path planning mechanism, it achieves automatic target localization and adaptive path optimization, significantly improving the intelligence level and inspection efficiency of the quality inspection robot.

[0006] A second objective of this invention is to provide a computer-readable storage medium.

[0007] The third objective of this invention is to provide a computer device.

[0008] The fourth objective of this invention is to propose a laser industrial quality inspection device based on semantic perception.

[0009] To achieve the above objectives, a first aspect of this invention proposes a laser industrial quality inspection method based on semantic perception. This method includes the following steps: acquiring multimodal data information of the industrial area to be inspected; performing spatiotemporal calibration on the multimodal data information to obtain a multidimensional feature vector for each fused sampling point; inputting the multidimensional feature vector of each sampling point into a pre-trained semantic segmentation network for semantic segmentation to generate a three-dimensional semantic map; acquiring detection instructions and parsing and inferring the detection instructions to obtain the coordinates of the detection target; performing path planning based on the detection target coordinates and the three-dimensional semantic map to obtain the optimal execution path to the target location; moving to the target location according to the optimal execution path and selecting the corresponding scanning mode based on the information in the three-dimensional semantic map to perform quality inspection on the target to obtain the corresponding detection result.

[0010] The laser industrial quality inspection method based on semantic perception provided by the present invention can construct a three-dimensional semantic map through laser and vision fusion technology to automatically identify equipment and inspection components; extract the spatial location information of the inspection target from human language commands using semantic reasoning algorithms; generate obstacle-avoiding, safe, and efficient inspection paths based on semantic perception path planning algorithms; realize the intelligentization of the entire process from "language command → target recognition → automatic detection"; and thus improve the intelligence level and inspection efficiency of the quality inspection robot.

[0011] In addition, the laser industrial quality inspection method based on semantic awareness proposed in the above embodiments of the present invention may also have the following additional technical features: Optionally, acquiring multimodal data information of the industrial area to be inspected includes: using a lidar to scan the spatial coordinates of each point in the industrial area to be inspected, and using a multi-layer scanning overlay and time fusion mechanism to obtain high-precision point cloud data; using an industrial camera to acquire image information of the industrial area to be inspected, wherein the image information includes grayscale, RGB color and specular / reflectance factor; using an IMU inertial measurement unit to measure the robot's three-axis acceleration and angular velocity in real time, and obtaining attitude and position estimates through integration.

[0012] Optionally, spatiotemporal calibration is performed on the multimodal data information to obtain a multidimensional feature vector for each fused sampling point. This includes: using three-dimensional joint calibration to achieve high-precision alignment of the lidar, camera, and IMU in a unified coordinate system to obtain point cloud or image spatial points in the world coordinate system; adaptively adjusting the point cloud weights using the specular / reflectance factor acquired by the industrial camera; and using multi-sensor interpolation correction to obtain a multidimensional feature vector for each fused sampling point.

[0013] Optionally, the multidimensional feature vector of each sampling point is input into a pre-trained semantic segmentation network for semantic segmentation to generate a three-dimensional semantic map. This includes: using a local attention mechanism to weighted integrate the spatial, color, and optical features of neighboring points to obtain enhanced local features; during point cloud semantic segmentation, using specular / reflectance factors acquired by an industrial camera to adaptively adjust the enhanced local features to obtain the segmentation result; and performing KNN semantic clustering on the segmented point cloud to obtain an object-level three-dimensional semantic map.

[0014] Optionally, the detection instructions are acquired and parsed and subjected to target inference to obtain the coordinates of the detected target. This includes: using a multi-layer self-attention mechanism of Transformer to capture the target category, spatial orientation, quantity, and detection task type in the detection instructions to obtain the instruction semantic vector; performing MLP mapping on the feature vector of each object in the 3D semantic map to obtain the object semantic representation; calculating the matching degree between the instruction semantic vector and the object semantic representation to obtain the preliminary target coordinates, and performing multi-level inference and confidence weighting to obtain the coordinates of the detected target.

[0015] Optionally, path planning is performed based on the detected target coordinates and the three-dimensional semantic map to obtain the optimal execution path to the target location, including: using a semantic-aware A algorithm to introduce semantic risk weights and environmental complexity constraints to obtain a node cost function; updating the three-dimensional semantic map in real time and dynamically adjusting the node cost function when the environment changes dynamically to obtain a preliminary path; and performing curve fitting and angle constraints on the preliminary path to minimize the change in turning angle and the difference in path length to obtain the optimal execution path to the target location.

[0016] Optionally, moving to the target location according to the optimal execution path and selecting a corresponding scanning mode based on the information of the three-dimensional semantic map to perform quality detection on the target and obtain the corresponding detection result includes: selecting a corresponding scanning mode based on the information of the three-dimensional semantic map to scan the target to obtain laser point cloud and reflection intensity data, wherein the scanning mode includes line scanning mode, area scanning mode and multi-angle scanning mode; fusing the laser point clouds obtained from scanning at different angles and in multiple modes to obtain a multi-dimensional feature vector; and using a defect identification algorithm to perform quality detection on the multi-dimensional feature vector to obtain the corresponding detection result.

[0017] To achieve the above objectives, a second aspect of the present invention provides a computer-readable storage medium storing a semantically aware laser industry quality inspection program thereon, which, when executed by a processor, implements the semantically aware laser industry quality inspection method as described above.

[0018] To achieve the above objectives, a third aspect of the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the semantic-aware laser industrial quality inspection method as described above.

[0019] To achieve the above objectives, a fourth aspect of the present invention proposes a laser industrial quality inspection device based on semantic perception, comprising: an acquisition module for acquiring multimodal data information of an industrial area to be inspected; a spatiotemporal calibration module for performing spatiotemporal calibration on the multimodal data information to obtain a multidimensional feature vector of each fused sampling point; a semantic mapping module for inputting the multidimensional feature vector of each sampling point into a pre-trained semantic segmentation network for semantic segmentation to generate a three-dimensional semantic map; a target reasoning module for acquiring detection instructions and parsing and reasoning about the detection instructions to obtain the coordinates of the detection target; a path planning module for performing path planning based on the detection target coordinates and the three-dimensional semantic map to obtain an optimal execution path to move to the target position; and a laser detection module for moving to the target position according to the optimal execution path and selecting a corresponding scanning mode to perform quality inspection on the target based on the information of the three-dimensional semantic map to obtain a corresponding detection result. Attached Figure Description

[0020] Figure 1 This is a schematic flowchart of a semantically perceptive laser industrial quality inspection method according to an embodiment of the present invention. Figure 2 This is a flowchart (system overall structure block diagram) of a semantically aware laser industrial quality inspection method according to an embodiment of the present invention. Figure 3 This is a schematic diagram illustrating the construction process of a three-dimensional semantic map according to an embodiment of the present invention; Figure 4 This is a flowchart illustrating the target reasoning process according to an embodiment of the present invention (target reasoning and semantic extraction flowchart). Figure 5 This is a flowchart illustrating semantic-aware path planning according to an embodiment of the present invention. Figure 6 This is a schematic diagram of the laser quality inspection execution and data visualization process according to an embodiment of the present invention; Figure 7 This is a block diagram of a semantically perceptive laser industrial quality inspection device according to an embodiment of the present invention. Detailed Implementation

[0021] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0022] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the invention to those skilled in the art.

[0023] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0024] refer to Figure 1 As shown, the semantic-aware laser industrial quality inspection method of this invention includes the following steps: S101, acquire multimodal data information of the industrial area to be inspected.

[0025] As an example, acquiring multimodal data information of the industrial area to be inspected includes: using a lidar to scan the spatial coordinates of each point in the industrial area to be inspected, and using a multi-layer scanning overlay and time fusion mechanism to obtain high-precision point cloud data; using an industrial camera to acquire image information of the industrial area to be inspected, wherein the image information includes grayscale, RGB color and specular / reflectance factor; using an IMU inertial measurement unit to measure the robot's three-axis acceleration and angular velocity in real time, and obtaining attitude and position estimates through integration.

[0026] Specifically, after the inspection robot enters the industrial area to be inspected, it first initiates a data acquisition process, using LiDAR, industrial cameras, and an IMU (Inertial Measurement Unit) to collect high-dimensional information about the entire scene. The LiDAR scans the target workpiece and its surrounding environment at a high frequency (e.g., 10–50 Hz), acquiring the spatial coordinates of each point.

[0027] in, This represents the total number of sampling points. To improve the sampling accuracy for small parts, welds, threaded holes, and highly reflective metal surfaces, the system employs a multi-layer scanning overlay and time fusion mechanism:

[0028] in, The increased number of superimposed frames significantly reduces measurement noise and laser reflection interference, enabling high-precision point cloud acquisition.

[0029] Industrial cameras simultaneously acquire high-resolution image information, including grayscale, RGB color, and specular / reflectance factors:

[0030] in, It represents the high light reflectance coefficient, which can be used for subsequent semantic segmentation and material type identification, thereby achieving optical feature compensation for metals and highly reflective surfaces.

[0031] The IMU (Inertial Measurement Unit) measures the robot's three-axis acceleration and angular velocity in real time, and obtains attitude and position estimates through integration.

[0032] in, Let be the rotation angle vector. It is a spatial position vector. Angular velocity, For speed, This represents the sampling time interval. IMU data is used for pose calibration and position compensation of point clouds and images, improving the reliability of data in moving or vibrating environments.

[0033] S102 performs spatiotemporal calibration on the multimodal data information to obtain the multidimensional feature vector of each fused sampling point.

[0034] As an example, spatiotemporal calibration is performed on multimodal data information to obtain a multidimensional feature vector for each fused sampling point. This includes: using three-dimensional joint calibration to achieve high-precision alignment of the LiDAR, camera, and IMU in a unified coordinate system to obtain point clouds or image spatial points in the world coordinate system; adaptively adjusting the point cloud weights using the specular / reflectance factor acquired by the industrial camera; and using multi-sensor interpolation correction to obtain a multidimensional feature vector for each fused sampling point.

[0035] Specifically, to achieve high-precision alignment of lidar, industrial cameras, and IMUs in a unified coordinate system, this application proposes a three-dimensional joint calibration method:

[0036] in, Points in the world coordinate system or image space. The extrinsic parameter matrix of the radar to the IMU, The extrinsic parameter matrix from the IMU to the camera. This provides the original coordinates for the camera or LiDAR. It achieves sub-centimeter alignment accuracy (≤5 mm), providing a reliable foundation for subsequent semantic mapping and target recognition.

[0037] For metal parts and highly reflective surfaces, this application further introduces a reflection intensity weighting matrix:

[0038] By adaptively adjusting the point cloud weights using the highlight information collected by the camera, the sampling accuracy of reflective areas is improved, effectively solving the impact of laser reflection and specular effect on the detection results.

[0039] For time synchronization, multi-sensor interpolation correction is used:

[0040] This mechanism can eliminate the effects of sensor acquisition delay and mechanical vibration, enabling high-precision fusion of point cloud, image, and attitude information.

[0041] Finally, a multidimensional feature vector is generated for each sampling point:

[0042] in, These are IMU acceleration and angle information, respectively. These multi-dimensional features can be simultaneously used for semantic mapping, target reasoning, and path planning, achieving high-precision mapping from spatial geometric information to semantic information, providing a solid data foundation for intelligent quality inspection throughout the entire process. Through this step, automatic fusion of laser point clouds, image textures, and IMU pose is achieved, solving problems such as the difficulty of sampling highly reflective metal surfaces, pose instability in dynamic scenes, and the difficulty of aligning multiple sensors. This provides high-precision and high-reliability input data for subsequent semantic understanding, target localization, and path planning.

[0043] S103 inputs the multidimensional feature vector of each sampling point into a pre-trained semantic segmentation network for semantic segmentation to generate a three-dimensional semantic map.

[0044] As an example, the multidimensional feature vector of each sampling point is input into a pre-trained semantic segmentation network for semantic segmentation to generate a three-dimensional semantic map. This includes: using a local attention mechanism to weight and integrate the spatial, color, and optical features of neighboring points to obtain enhanced local features; during the point cloud semantic segmentation process, using the specular / reflectance factor collected by an industrial camera to adaptively adjust the enhanced local features to obtain the segmentation result; and performing KNN semantic clustering on the segmented point cloud to obtain an object-level three-dimensional semantic map.

[0045] Specifically, such as Figure 3As shown, after completing high-precision data acquisition and spatiotemporal calibration, the fused point cloud and image features are input into an improved RandLA-Net deep network for semantic segmentation and feature extraction to construct a 3D map with rich semantic information. The implementation method is as follows: First, local feature enhancement is performed on the multidimensional feature vector of each point. An improved Local Attention Module (LAM) is used to weighted integrate the spatial, color, and optical features of neighboring points:

[0046] in, Point The neighborhood set, For feature similarity function, For attention weights, This enhances the local features. This mechanism improves the resolution of small parts, welds, threaded holes, and highly reflective surface features, enabling accurate identification of complex industrial components.

[0047] In the point cloud semantic segmentation process, considering that highly reflective metal surfaces may cause laser scattering and misclassification, this application introduces a reflectivity weighted compensation matrix. :

[0048] By utilizing the high-light reflection information collected by the camera to adaptively adjust local point features, the semantic segmentation error caused by reflection is effectively reduced, and the recognition accuracy of key targets such as welds and threaded holes is improved.

[0049] The segmented point cloud is further subjected to KNN semantic clustering to generate continuous semantic object instances:

[0050] in, For the m-th cluster instance, The threshold value is used for clustering neighborhood. KNN clustering groups points with the same semantic category into the same object instance, forming an object-level semantic map. This method ensures the integrity and spatial continuity of categories such as part bodies, welds, threaded holes, pipes, supports, hazardous areas, and passageways in the point cloud. The final generated semantic map data structure is as follows:

[0051] in, Let the coordinates be the points. For semantic category labels, The feature vectors are after local attention enhancement and reflection compensation. This semantic map not only contains spatial geometric information, but also integrates image texture, optical specular highlights, acceleration, and angle information, realizing intelligent mapping from geometric space to semantic space, and providing high-precision, multi-dimensional feature input for target reasoning and semantic perception path planning.

[0052] Through the above steps, this application achieves high-precision semantic mapping of complex parts and highly reflective surfaces in industrial environments, maintaining high robustness and high recognition accuracy in dynamic or complex environments, providing a reliable foundation for fully automated laser quality inspection.

[0053] S104: Obtain the detection command, parse the detection command and perform target inference to obtain the coordinates of the detection target.

[0054] As one embodiment, the detection instructions are obtained, and the detection instructions are parsed and the target inference is performed to obtain the detection target coordinates. This includes: using the multi-layer self-attention mechanism of Transformer to capture the target category, spatial orientation, quantity and detection task type in the detection instructions to obtain the instruction semantic vector; performing MLP mapping on the feature vector of each object in the 3D semantic map to obtain the object semantic representation; calculating the matching degree between the instruction semantic vector and the object semantic representation to obtain the preliminary target coordinates, and performing multi-level inference and confidence weighting to obtain the detection target coordinates.

[0055] Specifically, such as Figure 4 As shown, after completing semantic mapping, the system enters the instruction parsing and target reasoning stage. Operators can input detection instructions via natural language (speech or text), such as "Inspect the weld of the second flange on the left." The system first preprocesses the instructions, including word segmentation, part-of-speech tagging, named entity recognition, and syntactic parsing, to generate semantic embedding vectors.

[0056] in, Given the raw text sequence input by the operator, the Transformer network captures information such as target category, spatial location, quantity, and detection task type from the instructions through a multi-layer self-attention mechanism. Next, it extracts the feature vectors of each object from the semantic map. Mapped to the embedding space, forming a semantic representation of the object. :

[0057] Using semantic similarity matching functions The system calculates the matching degree between the instruction semantic vector and each object in the semantic map.

[0058]

[0059] in, These are the spatial coordinates of the target object in the 3D semantic map. The similarity function can be cosine similarity or weighted Euclidean distance combined with class weights.

[0060] in, The category matching weights ensure that category information in the instruction is matched first, while also taking into account spatial orientation and feature consistency.

[0061] To further improve the accuracy of target localization in complex environments, this application proposes a multi-level reasoning mechanism: First, a candidate object set is quickly filtered based on semantic categories. Then, spatial constraints and orientation information (such as "left side" and "bottom") are used to perform a secondary sorting of the candidate set to achieve precise localization. Simultaneously, for highly reflective or occluded areas, the system can use point cloud density, reflectivity weighting matrix, and image texture features for confidence weighting.

[0062] in, For the set of candidate objects, For local density weights of point clouds, It is a reflectivity weighting coefficient, which effectively improves the recognition accuracy of targets in complex industrial environments.

[0063] Through this step, the system can achieve intelligent understanding of natural language instructions, closely integrate semantic information with three-dimensional environmental data, realize semantic-level autonomous configuration of detection tasks, and enable robots to automatically identify and locate target objects without manual input of coordinates.

[0064] S105, perform path planning based on the detected target coordinates and the 3D semantic map to obtain the optimal execution path to move to the target location.

[0065] As an example, path planning is performed based on the detected target coordinates and a 3D semantic map to obtain the optimal execution path to the target location. This includes: using a semantic-aware A algorithm to introduce semantic risk weights and environmental complexity constraints to obtain a node cost function; updating the 3D semantic map in real time and dynamically adjusting the node cost function when the environment changes dynamically to obtain a preliminary path; and performing curve fitting and angle constraints on the preliminary path to minimize the change in turning angle and the difference in path length to obtain the optimal execution path to the target location.

[0066] Specifically, such as Figure 5 As shown, after completing target localization, the system uses the target object's coordinates... and 3D semantic maps Path planning is performed to generate the optimal movement path for the robot from its current position to the target position. It should be noted that traditional A* or Dijkstra algorithms only consider obstacles and distance factors, ignoring the semantic features of different regions, such as "dense equipment areas," "high-reflection danger areas," and "detection priority areas." This can lead to planned paths traversing high-risk areas, increasing detection errors and equipment wear. To improve safety, efficiency, and intelligence, this application proposes a semantically aware A* algorithm, which, based on traditional A* algorithms... Semantic risk weights and environmental complexity constraints are introduced on the basis of the algorithm.

[0067] First, the node cost function is defined as:

[0068] in, From the starting point to the node The actual cost, To estimate the cost for the heuristic function, For nodes The risk value of the semantic region (e.g., high temperature zone, electrical zone, dangerous zone). The complexity of the environment surrounding the node (obstacle density or spatial confinement). and These are adjustable weight parameters. By simultaneously considering semantic risk and space complexity, the system can generate safer, smoother, and computationally efficient paths.

[0069] Secondly, to enhance the dynamic adaptability of path planning, this application introduces a real-time semantic update mechanism: when the robot detects new obstacles or semantic changes during its movement (such as the addition of new mobile devices or the entry of workers into the area), the map information... It will be updated in real time, and the node cost function will be dynamically adjusted:

[0070] in, and For a moment The real-time semantic risk and complexity values ​​are used to achieve dynamic obstacle avoidance and path replanning. To improve path smoothness and reduce frequent turning of the robotic arm or mobile platform, this application adopts a multi-node curve optimization mechanism to perform curve fitting and angle constraints on the initially generated path:

[0071] in, For changes in the turning angle of adjacent nodes, This represents the difference in path length between nodes. This mechanism ensures smooth path movement, reducing energy consumption and collision risks during robot motion.

[0072] Furthermore, path planning, combined with detection task priority, schedules multi-target detection tasks: prioritizing access to high-risk or urgent detection areas while considering the shortest path and minimum energy consumption, thus forming an optimized task scheduling strategy.

[0073] in, For the first The detection priority weights of each target are determined, and the optimal path and task are achieved through comprehensive optimization.

[0074] Through the above steps, this application can achieve intelligent navigation from the current position to the detection target in complex industrial environments. It can not only automatically avoid dangerous areas and obstacles, but also take into account detection efficiency, path smoothness and safety, thus ensuring the high-precision execution of subsequent laser detection.

[0075] S106, Move to the target position according to the optimal execution path, and select the corresponding scanning mode to perform quality detection on the target based on the information of the three-dimensional semantic map, so as to obtain the corresponding detection results.

[0076] As an example, the target location is moved according to the optimal execution path, and the target is quality inspected by selecting the corresponding scanning mode based on the information of the 3D semantic map to obtain the corresponding detection result. This includes: selecting the corresponding scanning mode to scan the target based on the information of the 3D semantic map to obtain laser point cloud and reflection intensity data, wherein the scanning mode includes line scanning mode, area scanning mode and multi-angle scanning mode; fusing the laser point clouds obtained from scanning at different angles and in multiple modes to obtain a multi-dimensional feature vector; and using a defect identification algorithm to perform quality inspection on the multi-dimensional feature vector to obtain the corresponding detection result.

[0077] Specifically, such as Figure 6 As shown, once the robot reaches the target location along the planned path, the laser detection module adaptively selects the most suitable scanning mode based on the target object's category, surface characteristics, and orientation information in the 3D semantic map, ensuring high-precision and high-efficiency detection. The system dynamically adjusts laser scanning parameters, including scanning speed, resolution, angle, laser power, and scanning mode (line scan, area scan, or multi-angle scan), according to different workpiece types and detection tasks, forming a complete task adaptive execution mechanism.

[0078] For slender structures such as welds, the system employs a line scanning mode, sampling at high density along the weld direction, while combining point cloud overlay and filtering algorithms.

[0079] It enables accurate identification of cracks, holes, and minute defects.

[0080] For planar parts or large-area metal surfaces, a surface scanning mode is used, which utilizes multi-layer scanning trajectories and adaptive laser power adjustment to compensate for surface reflection and texture differences.

[0081] For complex structures such as threaded holes, supports, or pipe interfaces, the system combines multi-angle scanning and rotation / tilt adjustment strategies to achieve comprehensive coverage of hidden surfaces or complex curved areas. The scanning path can be dynamically optimized within a local range, combined with a reflective intensity weighted matrix. With point cloud density information :

[0082] Improve sampling accuracy in reflective and recessed areas while reducing noise interference.

[0083] Furthermore, the laser detection module of this application possesses real-time data fusion capabilities, enabling the fusion of point clouds obtained from scanning at different angles and in multiple modes to generate a high-density three-dimensional point cloud model for defect identification and surface feature analysis. The system extracts features from the scanned point cloud, including curvature, normal vector, texture intensity, and reflection characteristics, to form a multi-dimensional feature vector:

[0084] in, For curvature, It is the normal vector. These are semantic category tags used to accurately locate various defect types such as cracks, dents, scratches, and burrs.

[0085] The laser scanning process is also linked with the path planning module, adjusting the scanning sequence and angle in real time to prioritize the inspection of high-risk areas or critical components, ensuring high-precision inspection of high-value parts within a limited time. Simultaneously, the system can dynamically adjust scanning parameters based on changes in the on-site environment, such as minor robot shifts, vibrations, or changes in lighting, ensuring inspection accuracy and robustness.

[0086] Finally, the scanned data is transmitted in real time to the results analysis module, where it is correlated with the semantic map to achieve defect localization, visual annotation, and automatic generation of quality reports. The inspection results are not only fed back into the system database but can also be used for self-learning optimization, gradually improving the laser inspection strategy and task adaptability, thus achieving an intelligent closed loop in industrial quality inspection.

[0087] As a specific example, such as Figure 2As shown, the system performs result analysis and feedback after laser detection: After laser scanning is completed, the system transmits the collected high-density point cloud data to the result analysis module. This module first spatially aligns the scanned data with a pre-generated 3D semantic map, and achieves precise localization of the detection data through multi-dimensional feature matching and semantic label association. The system combines point cloud curvature, normal vector, reflection intensity, and texture features to construct a multi-dimensional description vector for each detection point:

[0088] in, This system represents the confidence score for defects, classifying and grading different defect types such as cracks, dents, scratches, and burrs through statistical analysis and machine learning algorithms. Combined with semantic map information, it provides semantic-level visual annotations of the inspected objects, generating a complete quality report. The annotations include the defect location, type, size, severity, and its semantic region (e.g., weld, flange, or pipe joint), facilitating quick quality assessment by operators. The report also supports 3D visualization, overlaying data from different scanning angles to create a three-dimensional representation of the defects, enhancing the intuitiveness and operability of the inspection results.

[0089] This application implements a closed-loop feedback mechanism between detection results and semantic maps. The system not only outputs detection reports but also sends defect information back to the semantic mapping and target inference modules to update object states and semantic labels, enhancing the accuracy and efficiency of subsequent detection tasks. For example, for repeated detection of the same part, the system can adjust the laser scanning path, scanning density, and scanning angle based on historical detection results to achieve adaptive optimization. Simultaneously, the result analysis module can link with the path planning and laser execution modules to dynamically optimize the detection strategy. When high-risk or high-defect-density areas are detected, the system can replan the scanning path, increase sampling density, or adjust the scanning angle to ensure comprehensive detection of critical components and hazardous areas.

[0090] In summary, this application aims to address several key issues prevalent in existing laser industrial quality inspection technologies, including the high dependence of inspection coordinates on manual input, the lack of semantic understanding in navigation paths, and insufficient intelligence in the inspection process. This invention introduces core technologies such as semantic mapping, target reasoning, and semantic-aware path planning, enabling the inspection robot to autonomously identify inspection objects, intelligently understand operational instructions, and plan safe and efficient inspection paths in complex industrial environments. This achieves a closed-loop automated process from "language command, target recognition, and laser inspection execution," significantly improving the efficiency and reliability of industrial quality inspection. The core lies in constructing a semantically enhanced 3D environmental perception and intelligent decision-making system. The system utilizes an improved RandLA-Net deep network combined with the KNN semantic clustering algorithm to perform real-time segmentation and semantic annotation of laser point cloud data, generating a high-precision 3D semantic map. It organically integrates geometric spatial information with semantic information, enabling the robot to accurately distinguish various industrial elements such as the main body of the part, welds, threaded holes, pipes, supports, and hazardous areas. This semantically enhanced mapping mechanism not only significantly improves the ability to identify complex parts, small components, and highly reflective surface features, but also provides a reliable foundation for subsequent target localization and path planning, ensuring detection accuracy and stability under complex assembly environments and multi-target task conditions.

[0091] In terms of target recognition and natural language command parsing, this invention proposes an object instance reasoning algorithm that can intelligently match the detection commands input by operators via voice or text with the feature information of each object in a semantic map, thereby achieving automatic localization of the detection object. It can parse the target category, spatial orientation, and detection task type contained in the command, thus completing the semantic-level autonomous configuration of the detection task, enabling the robot to accurately execute complex detection tasks without manual coordinate input. Through this mechanism, operators only need to provide human-like commands, such as "inspect the weld of the second flange on the left," and the system can automatically understand and locate the detection target, improving the flexibility and operational convenience of industrial quality inspection. In terms of navigation and path planning, this invention adopts a semantic-aware path planning algorithm, introducing semantic weights, task priorities, and environmental risk assessment based on an improved A* search strategy, enabling the robot to move safely, efficiently, and smoothly in complex industrial environments. During path planning, the system not only considers geometric obstacles but also comprehensively assesses the risk level of each semantic region, such as high-temperature areas, electrical areas, and areas with dense equipment, thereby effectively reducing potential hazards. In addition, path planning can be dynamically adjusted to automatically avoid obstacles and cluttered areas, improving the robustness and detection efficiency of inspection tasks, while ensuring the robot's collaborative execution capability in multi-objective and multi-task scenarios.

[0092] This invention also proposes an adaptive execution mechanism for inspection tasks, which dynamically selects the laser scanning mode and inspection sequence based on the semantic category and spatial location of the target object, enabling precise multi-angle and multi-point scanning of complex component surfaces. Through point cloud fusion, feature extraction, and defect analysis, it can efficiently identify various industrial defect types such as weld cracks, threaded hole defects, and scratches on planar parts, and generate high-precision inspection results. Simultaneously, the inspection data is correlated with the semantic map in real time, and the location, type, and severity of defects are displayed through visual annotations, achieving closed-loop management of industrial quality inspection information. This closed-loop mechanism not only enables the system to generate complete and structured quality reports but also feeds inspection experience back to the semantic mapping and target reasoning modules, continuously optimizing inspection accuracy and path planning strategies, and gradually improving the robot's self-learning and adaptability in complex industrial environments.

[0093] Furthermore, it has broad applicability, suitable for automated inspection of welded parts, threaded holes, pipes, supports, highly reflective metal components, and multi-target assembly environments, exhibiting significant advantages, especially in complex assembly conditions, highly reflective surfaces, and multi-target inspection tasks. Through the organic combination of semantic task execution, intelligent path planning, and adaptive laser detection, this invention achieves high-precision, high-security, high-autonomy, and high-intelligence industrial quality inspection, effectively reducing manual intervention and improving inspection efficiency and reliability. It provides a technologically advanced and widely applicable complete solution for the industrial quality inspection field, possessing significant technical advantages and practical value.

[0094] The advantages of this invention are mainly reflected in the following aspects. First, it enables semantic and automated detection tasks, fundamentally reducing reliance on manual coordinate input and operation, minimizing human error, and improving operational efficiency. Second, through semantically enhanced 3D mapping and target instance reasoning technology, the robot can accurately identify complex parts and highly reflective surface features, achieving high-precision detection in multiple targets and scenarios. Third, semantically perceptive path planning can generate safe, efficient, and smooth detection paths based on environmental risks and task priorities, improving the reliability and safety of industrial quality inspection. Simultaneously, the adaptive laser detection mechanism and data closed-loop management can automatically adjust scanning strategies for different targets, enabling multi-angle and multi-point detection of complex parts, and feeding the detection results back to the system to optimize target recognition and path planning strategies, achieving continuous learning and intelligent optimization.

[0095] In addition, the present invention also proposes a computer-readable storage medium storing a semantically aware laser industrial quality inspection program, which, when executed by a processor, implements the semantically aware laser industrial quality inspection method as described above.

[0096] In addition, this invention also proposes a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the semantic-aware laser industrial quality inspection method described above.

[0097] Figure 7 This is a block diagram of a semantically aware laser industrial quality inspection device according to an embodiment of the present invention. Figure 7 As shown, the detection device includes: an acquisition module 10, a spatiotemporal calibration module 20, a semantic mapping module 30, a target reasoning module 40, a path planning module 50, and a laser detection module 60; The system comprises the following modules: Acquisition module 10 acquires multimodal data information of the industrial area to be inspected; Spatiotemporal calibration module 20 performs spatiotemporal calibration on the multimodal data information to obtain the multidimensional feature vector of each fused sampling point; Semantic mapping module 30 inputs the multidimensional feature vector of each sampling point into a pre-trained semantic segmentation network for semantic segmentation to generate a three-dimensional semantic map; Target inference module 40 acquires detection instructions and performs parsing and target inference on the detection instructions to obtain the coordinates of the detection target; Path planning module 50 performs path planning based on the detection target coordinates and the three-dimensional semantic map to obtain the optimal execution path to move to the target position; and Laser detection module 60 moves to the target position according to the optimal execution path and selects the corresponding scanning mode based on the information of the three-dimensional semantic map to perform quality detection on the target to obtain the corresponding detection result.

[0098] It should be noted that the foregoing explanation of the embodiment of the semantic-aware laser industrial quality inspection method also applies to the semantic-aware laser industrial quality inspection device of this embodiment, and will not be repeated here.

[0099] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0100] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0101] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0102] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0103] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0104] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0105] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0106] In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0107] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0108] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0109] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms should not be construed as necessarily referring to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0110] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A laser industrial quality inspection method based on semantic perception, characterized in that, Includes the following steps: Acquire multimodal data information of the industrial area to be inspected; Spatiotemporal calibration is performed on the multimodal data information to obtain a multidimensional feature vector for each fused sampling point; The multidimensional feature vector of each sampling point is input into a pre-trained semantic segmentation network for semantic segmentation to generate a three-dimensional semantic map; Obtain the detection command, and parse and infer the target from the detection command to obtain the coordinates of the detection target; Path planning is performed based on the detected target coordinates and the three-dimensional semantic map to obtain the optimal execution path to move to the target location; The system moves to the target location according to the optimal execution path and selects the corresponding scanning mode to perform quality detection on the target based on the information of the three-dimensional semantic map, so as to obtain the corresponding detection results.

2. The laser industrial quality inspection method based on semantic perception as described in claim 1, characterized in that, Acquire multimodal data information of the industrial area to be inspected, including: The spatial coordinates of each point in the industrial area to be inspected are scanned using lidar, and a multi-layer scanning overlay and time fusion mechanism is used to obtain high-precision point cloud data. An industrial camera is used to acquire image information of the industrial area to be inspected, wherein the image information includes grayscale, RGB color, and specular / reflectance factor; The robot's three-axis acceleration and angular velocity are measured in real time using an IMU (Inertial Measurement Unit), and attitude and position estimates are obtained through integration.

3. The laser industrial quality inspection method based on semantic perception as described in claim 2, characterized in that, The multimodal data information is spatiotemporally calibrated to obtain a multidimensional feature vector for each fused sampling point, including: Three-dimensional joint calibration is used to achieve high-precision alignment of lidar, camera and IMU in a unified coordinate system to obtain point cloud or image space points in the world coordinate system; The point cloud weights are adaptively adjusted using the specular / reflectance factor acquired by an industrial camera, and multi-sensor interpolation correction is used to obtain the multi-dimensional feature vector of each fused sampling point.

4. The laser industrial quality inspection method based on semantic perception as described in claim 3, characterized in that, The multidimensional feature vector of each sampling point is input into a pre-trained semantic segmentation network for semantic segmentation to generate a three-dimensional semantic map, including: A local attention mechanism is used to weight and integrate the spatial, color, and optical features of neighboring points to obtain enhanced local features; In the point cloud semantic segmentation process, the specular / reflectance factor acquired by an industrial camera is used to adaptively adjust the enhanced local features in order to obtain the segmentation result; KNN semantic clustering is performed on the segmented point cloud to obtain an object-level 3D semantic map.

5. The laser industrial quality inspection method based on semantic perception as described in claim 4, characterized in that, Acquire detection instructions, and parse and infer the target from the detection instructions to obtain the coordinates of the detected target, including: A multi-layer self-attention mechanism of Transformer is used to capture the target category, spatial orientation, quantity and detection task type in the detection instructions to obtain the instruction semantic vector; MLP mapping is performed on the feature vector of each object in the 3D semantic map to obtain the semantic representation of the object; The matching degree between the instruction semantic vector and the object semantic representation is calculated to obtain the preliminary target coordinates, and multi-level reasoning and credibility weighting are performed to obtain the detection target coordinates.

6. The laser industrial quality inspection method based on semantic perception as described in claim 5, characterized in that, Path planning is performed based on the detected target coordinates and the 3D semantic map to obtain the optimal execution path to the target location, including: The semantic-aware A algorithm is used to introduce semantic risk weights and environmental complexity constraints to obtain the node cost function; When the environment changes dynamically, the 3D semantic map is updated in real time, and the node cost function is dynamically adjusted to obtain the initial path; Curve fitting and angle constraints are applied to the initial path to minimize the change in steering angle and the difference in path length, thereby obtaining the optimal execution path to move to the target position.

7. The laser industrial quality inspection method based on semantic perception as described in claim 6, characterized in that, The target location is moved according to the optimal execution path, and the corresponding scanning mode is selected based on the information from the 3D semantic map to perform quality detection on the target, so as to obtain the corresponding detection results, including: The target is scanned by selecting the corresponding scanning mode based on the information of the three-dimensional semantic map to obtain laser point cloud and reflection intensity data. The scanning modes include line scanning mode, area scanning mode and multi-angle scanning mode. Laser point clouds obtained from scanning at different angles and in multiple modes are fused to obtain multidimensional feature vectors; A defect identification algorithm is used to perform quality detection on the multidimensional feature vector to obtain the corresponding detection results.

8. A computer-readable storage medium, characterized in that, It stores a semantically aware laser industry quality inspection program, which, when executed by a processor, implements the semantically aware laser industry quality inspection method as described in any one of claims 1-7.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the semantic-aware laser industrial quality inspection method as described in any one of claims 1-7.

10. A laser industrial quality inspection device based on semantic perception, characterized in that, include: The acquisition module is used to acquire multimodal data information of the industrial area to be inspected; The spatiotemporal calibration module is used to perform spatiotemporal calibration on the multimodal data information to obtain the multidimensional feature vector of each fused sampling point; The semantic mapping module is used to input the multidimensional feature vector of each sampling point into a pre-trained semantic segmentation network for semantic segmentation to generate a three-dimensional semantic map. The target inference module is used to acquire detection instructions and parse and infer the detection instructions to obtain the coordinates of the detection target; The path planning module is used to perform path planning based on the detected target coordinates and the three-dimensional semantic map to obtain the optimal execution path to move to the target location; The laser detection module is used to move to the target position according to the optimal execution path, and select the corresponding scanning mode according to the information of the three-dimensional semantic map to perform quality detection on the target in order to obtain the corresponding detection result.

Images