Workpiece visual identification system for intelligent factory

Abstract

The invention relates to the technical field of machine vision, and particularly discloses a workpiece visual identification system for an intelligent factory. The system comprises a multi-modal data acquisition module, a feature extraction and fusion module, a three-dimensional space reconstruction module, a dynamic identification decision-making module and a system control and output module. And high-precision and high-reliability identification and positioning of the identity and the pose of the workpiece are realized.

Description

Technical Field

[0001] This invention belongs to the field of machine vision technology, and specifically relates to a workpiece visual recognition system for smart factories. Background Technology

[0002] In the fields of intelligent manufacturing and industrial automation, visual recognition technology is one of the key supporting technologies for realizing intelligent and unmanned operation of production processes, and it is widely used in core aspects such as workpiece positioning, detection, classification, and guidance. Among them, the workpiece visual recognition system, as the sensing front end connecting the physical world and the digital control system, directly determines the efficiency and reliability of the entire production system through its recognition accuracy and robustness.

[0003] Existing technologies typically rely on single two-dimensional image information for workpiece recognition and positioning. However, in actual factory environments, lighting conditions may change abruptly, workpiece surfaces are prone to reflection or oil stains, and workpieces often present complex situations such as stacking, partial obstruction, and changing postures on material frames or conveyor belts.

[0004] These interfering factors make it difficult for traditional vision systems to reliably extract image features, resulting in frequent misidentification or missed identification, which seriously affects production cycle and quality control. Therefore, there is an urgent need for a workpiece vision recognition system that can adapt to complex industrial environments and has higher recognition accuracy and robustness. Summary of the Invention

[0005] The present invention aims to provide a workpiece visual recognition system for smart factories, in order to solve the problems of low recognition accuracy and poor robustness in complex industrial environments caused by reliance on single two-dimensional image information in the prior art.

[0006] This invention provides an intelligent visual recognition system based on multimodal information fusion and hierarchical decision-making. By integrating multiple sensors and introducing collaborative perception and confidence assessment mechanisms, the system achieves accurate and stable recognition of workpieces under complex working conditions. Specifically, the system includes a multimodal data acquisition module, a feature extraction and fusion module, a three-dimensional spatial reconstruction module, a dynamic recognition decision-making module, and a system control and output module.

[0007] The multimodal data acquisition module is responsible for synchronously acquiring multi-source sensing data of the workpiece. This module is equipped with a high-resolution industrial area scan camera, a line laser 3D scanner, and a near-infrared depth sensor. The industrial area scan camera acquires two-dimensional color images of the workpiece at a rate of fps. The line laser 3D scanner scans the workpiece surface with a vertical resolution of 0.1 mm, obtaining high-precision 3D point cloud data. The near-infrared depth sensor simultaneously acquires depth image information of the workpiece. All sensors achieve microsecond-level time synchronization through hardware trigger signals and are aggregated through a unified data interface.

[0008] The feature extraction and fusion module processes the acquired multi-source data and generates fused features. This module includes a 2D image feature extraction unit, a 3D geometric feature extraction unit, and a multimodal feature fusion unit. The 2D image feature extraction unit uses a deep convolutional neural network to process color images, extracting the texture, contour, and color features of the workpiece. The 3D geometric feature extraction unit preprocesses the 3D point cloud data, including denoising, downsampling, and normal vector calculation, and then extracts the surface curvature, edge lines, and key point features of the workpiece.

[0009] The multimodal feature fusion unit receives the outputs from the two-dimensional and three-dimensional feature extraction units, maps the features of different modalities to a unified feature space through a designed feature alignment network, and performs weighted concatenation to form the final fused feature vector.

[0010] The 3D spatial reconstruction module accurately reconstructs the 3D model of the workpiece in physical space and determines its pose based on the fused features and the original 3D data. This module first uses an iterative nearest-point algorithm to coarsely register the 3D point cloud of the current frame with a pre-stored standard workpiece model. Then, combining the semantic information contained in the fused feature vectors, a feature-based point cloud registration method is used for fine registration, thereby calculating the workpiece's 3D translation vector and rotation matrix relative to the world coordinate system, i.e., its six-degree-of-freedom pose.

[0011] The dynamic identification and decision-making module is the core of this system, which makes the final identity and status determination based on the output of the preceding modules. This module includes a confidence assessment unit and a decision arbitration unit.

[0012] The confidence evaluation unit receives in parallel the two-dimensional recognition confidence score from the two-dimensional image feature extraction unit and the three-dimensional matching score from the three-dimensional space reconstruction module.

[0013] The confidence score for two-dimensional recognition is obtained by calibrating the output of the Softmax layer of a deep convolutional neural network.

[0014] The three-dimensional matching score is mapped from the average matching error after fine registration to the range of 0 to 1 via an inverse proportional function.

[0015] The decision arbitration unit presets a dynamic decision threshold, which is adaptively adjusted based on the ambient light sensor readings and historical recognition success rates.

[0016] When the maximum value of the two-dimensional recognition confidence score and the three-dimensional matching score is higher than the dynamic decision threshold, the system adopts the recognition result corresponding to the higher score.

[0017] When both scores are below the dynamic decision threshold and the difference between them is less than the preset tolerance range, the decision arbitration unit initiates a weighted fusion judgment, weights the two scores according to their historical accuracy, and uses the weighted score as the basis for the final decision.

[0018] The system control and output module is responsible for coordinating the timing of various modules within the system and converting the final decision of the dynamic identification and decision module into an industrial bus protocol signal for output to downstream actuators, such as robot controllers or programmable logic controllers.

[0019] In a preferred embodiment of the present invention, the line laser 3D scanner and the near-infrared depth sensor in the multimodal data acquisition module are orthogonally arranged in space to ensure that the workpiece is covered from different perspectives and to minimize the detection blind zone.

[0020] Furthermore, the feature alignment network used in the feature extraction and fusion module is a multilayer perceptron with three fully connected layers, which learns the optimal mapping relationship from heterogeneous feature space to unified feature space through a large number of samples.

[0021] Furthermore, the adaptive adjustment process of the dynamic decision threshold in the dynamic recognition decision module is as follows: the system continuously monitors the ambient light intensity. When the light intensity is below 300 lux or above 1500 lux, the dynamic decision threshold is increased by 0.1 based on the baseline value of 0.85 to enhance the strictness of the recognition standard. Simultaneously, the system records the success rate of the past 100 recognition tasks. If the success rate is below 95%, the dynamic decision threshold is further increased by 0.05 based on the light intensity adjustment.

[0022] Furthermore, in the point cloud fine registration stage, the 3D space reconstruction module introduces semantic information from the fused feature vectors as a soft constraint for registration. Specifically, feature similarity is used as a term in the loss function and optimized together with the point-to-point distance, thereby improving the robustness of registration under partial occlusion.

[0023] Furthermore, the system control and output module supports mainstream industrial Ethernet protocols. Its output data packets not only contain the workpiece's identification code and six-degree-of-freedom pose information, but also include the overall confidence score of this identification, which is used by downstream systems to make reliability judgments.

[0024] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0025] 1. This invention overcomes the limitations of single visual information in complex industrial environments by constructing a framework for multimodal data acquisition and deep fusion.

[0026] 2. The system complements the rich texture and color information of two-dimensional images with the precise geometric spatial information of three-dimensional point clouds, thereby improving the completeness and robustness of feature representation.

[0027] 3. The introduced dynamic identification decision mechanism, through confidence assessment and adaptive threshold arbitration, realizes intelligent selection and optimization of identification strategies under different working conditions, avoiding misjudgment caused by the failure of a single information source.

[0028] 4. The 3D spatial reconstruction module combines semantic features for fine registration, which addresses the issues of partial occlusion and varied poses of the workpiece, ensuring the accuracy of pose estimation.

[0029] 5. The entire system architecture is clearly structured, and the modules work together efficiently. Ultimately, it achieves high-precision and high-reliability identification and positioning of workpieces under challenging scenarios such as changes in lighting, surface interference, and stacking obstruction, providing a solid technical guarantee for the stable and efficient operation of the smart factory. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the overall technical solution architecture of the workpiece visual recognition system for smart factories proposed in this invention;

[0031] Figure 2 This is a schematic diagram of the core principle framework of multimodal information fusion and hierarchical decision-making in this invention;

[0032] Figure 3 This is a logical flowchart of the multimodal data acquisition and collaborative sensing in this invention;

[0033] Figure 4 This is a schematic diagram of the multi-level interaction relationship and data flow of the feature extraction and fusion module in this invention;

[0034] Figure 5 This is a framework diagram of the confidence assessment and arbitration mechanism of the dynamic identification decision module in this invention; Detailed Implementation

[0035] Please refer to the attached document. Figure 1 This embodiment details the specific technical implementation of a workpiece visual recognition system for smart factories. The system aims to address the problems of low recognition accuracy and poor robustness caused by reliance on single two-dimensional image information in complex industrial environments by integrating multiple sensors and employing multimodal information fusion and hierarchical decision-making mechanisms. The overall system architecture strictly adheres to modular design principles, with modules interacting through clearly defined data interfaces and communication protocols to ensure efficient and reliable information flow.

[0036] Please refer to the attached document. Figure 3The multimodal data acquisition module is the data source for the entire system, responsible for synchronously acquiring multi-source sensing data of the workpiece. This module physically comprises three core sensor components: a high-resolution industrial area scan camera, a line laser 3D scanner, and a near-infrared depth sensor.

[0037] The industrial area scan camera uses global shutter technology to continuously acquire two-dimensional color images of the workpiece at a fixed rate of fps. Its image resolution is set to 4096 pixels × 2160 pixels, and the color depth is 24 bits.

[0038] The line laser 3D scanner uses the principle of triangulation. Its laser linewidth is precisely optically designed, enabling it to scan the workpiece surface at high speed with a longitudinal resolution of 0.1mm, generating dense point cloud data containing millions of 3D coordinate points.

[0039] The near-infrared depth sensor uses the time-of-flight method. It emits modulated near-infrared light and measures the phase difference of its reflection to simultaneously acquire the depth image information of the workpiece. The resolution of the depth image is 1024 pixels × 768 pixels, and the depth measurement accuracy reaches ±1mm.

[0040] To achieve strict alignment of data in the time dimension, all sensors receive a synchronization signal generated by the same hardware trigger circuit. This signal has a period of 33.3ms and a pulse width of 1μs, ensuring that the time deviation of all data acquisition actions is controlled at the microsecond level.

[0041] The collected raw data is aggregated through a unified gigabit Ethernet interface. The data format follows a predefined industrial image transmission protocol, which specifies that the header information of the image data includes frame number, timestamp, sensor identifier, and data check code.

[0042] In terms of sensor spatial layout, as a preferred implementation, the line laser 3D scanner and the near-infrared depth sensor are orthogonally arranged and installed in terms of mechanical structure.

[0043] Specifically, the line laser 3D scanner is fixed directly above the workpiece conveyor belt, with its laser projection plane perpendicular to the conveyor belt plane.

[0044] The near-infrared depth sensor is installed on the side of the workpiece conveyor belt, with its optical axis parallel to the direction of the conveyor belt and forming a 90° angle with the scanning plane of the line laser 3D scanner.

[0045] This orthogonal layout ensures that three-dimensional information of the workpiece is captured from two distinct perspectives, minimizing blind spots caused by the workpiece's geometry or its orientation on the conveyor belt.

[0046] The installation location of industrial area scan cameras takes into account both the overall field of view and lighting conditions. They are usually placed near line laser 3D scanners and equipped with ring LED light sources to provide stable and uniform illumination.

[0047] Please refer to the attached document. Figure 4 The feature extraction and fusion module is responsible for performing in-depth processing on the raw data collected by the multimodal data acquisition module and generating a fusion feature representation with complementary information.

[0048] At the software level, this module is divided into three functionally interconnected sub-units: a two-dimensional image feature extraction unit, a three-dimensional geometric feature extraction unit, and a multimodal feature fusion unit.

[0049] The 2D image feature extraction unit is specifically designed for processing 2D color images from industrial area scan cameras. This unit incorporates a pre-trained deep convolutional neural network model, whose architecture includes five convolutional layers, three pooling layers, and two fully connected layers.

[0050] The input image is first normalized to 256 pixels × 256 pixels, and pixel value normalization is performed. The convolutional layer uses a 3 × 3 convolutional kernel with a stride of 1 and a rectified linear unit as the activation function.

[0051] The pooling layer uses max pooling with a window size of 2×2.

[0052] This deep convolutional neural network is trained on a large dataset of labeled workpiece images to learn and extract deep features that can distinguish different workpiece categories. These features mainly include the workpiece's micro-texture patterns, macro-contour shapes, and specific color distribution statistics.

[0053] The output of this unit is a 2048-dimensional floating-point feature vector that highly summarizes the visual semantic information of the input image.

[0054] The 3D geometric feature extraction unit focuses on processing 3D point cloud data from a line laser 3D scanner.

[0055] The execution flow of this unit includes a series of ordered geometric data processing steps. The first step is point cloud preprocessing, which specifically includes denoising the original point cloud using a statistical outlier removal algorithm. This algorithm calculates the average distance between each point and its 50 nearest neighbors and removes points whose distance exceeds one standard deviation from the global average.

[0056] Subsequently, a voxel grid downsampling method was used to reduce the point cloud density to 20% of the original density in order to balance computational efficiency and feature fidelity.

[0057] After preprocessing, the normal vector of each remaining point is calculated. The normal vector estimation is based on the eigenvector corresponding to the minimum eigenvalue of the covariance matrix of all points in a neighborhood with a radius of 2 mm around that point.

[0058] During the feature calculation stage, this unit extracts a variety of three-dimensional geometric feature descriptors, including but not limited to surface curvature values ​​derived from feature values ​​calculated based on the point neighborhood covariance matrix, workpiece boundary line features extracted by the edge point detection algorithm, and key point features identified using the intrinsic shape signature method.

[0059] Ultimately, the unit outputs a 1024-dimensional geometric feature vector.

[0060] The multimodal feature fusion unit is the core hub connecting the two-dimensional and three-dimensional feature domains.

[0061] This unit receives a 2048-dimensional feature vector from the 2D image feature extraction unit and a 1024-dimensional feature vector from the 3D geometric feature extraction unit.

[0062] Since these two feature vectors originate from different modalities and physical spaces, their numerical distributions and semantic levels differ, making direct splicing or simple fusion ineffective.

[0063] Therefore, this system uses a specially designed feature alignment network to achieve unified mapping of heterogeneous feature spaces.

[0064] The feature alignment network is a multilayer perceptron with three fully connected layers.

[0065] Its input layer has 3072 nodes, corresponding to the concatenation of two input feature vectors. The first fully connected layer has 1024 nodes, the second fully connected layer has 512 nodes, and the output layer has 256 nodes.

[0066] All hidden layers use the modified linear unit activation function, and the output layer uses the hyperbolic tangent activation function to constrain the feature values ​​to the range of -1 to +1.

[0067] The network is trained using a large number of samples containing paired 2D images and 3D point clouds. Its learning objective is to minimize the Euclidean distance between different modal features from the same workpiece in a unified 256-dimensional feature space after being mapped by the network, while maximizing the feature distance between different workpieces.

[0068] The training process uses a stochastic gradient descent optimizer, with the learning rate initialized to 0.001, and a contrastive loss function.

[0069] Once the network training converges, it learns the optimal mapping relationship from the original heterogeneous feature space to the unified feature space.

[0070] During the actual inference phase, the two input feature vectors are concatenated and fed into the feature alignment network, which outputs a 256-dimensional fused feature vector.

[0071] This fused feature vector integrates the visual appearance information and three-dimensional geometric structure information of the workpiece, providing a more robust feature representation for subsequent recognition and localization tasks.

[0072] The core task of the 3D space reconstruction module is to accurately reconstruct the 3D model of the workpiece in the physical world based on the fused feature vector output by the multimodal feature fusion module and the original 3D point cloud data, and to calculate its precise six-degree-of-freedom pose.

[0073] Please refer to the attached document. Figure 2 The execution process of this module is divided into two main stages: coarse registration and fine registration.

[0074] The coarse registration stage employs the iterative nearest-point algorithm. This algorithm uses the preprocessed 3D point cloud of the current frame as the source point cloud and the pre-stored standard workpiece 3D model point cloud as the target point cloud. During algorithm initialization, an initial rigid body transformation estimate is provided for the source point cloud based on the workpiece's prior approximate position or using the coarse classification results based on fused feature vectors.

[0075] Subsequently, the iterative nearest point algorithm begins to iteratively execute two core steps: the first step is the nearest point search, which finds the point in the target point cloud with the closest Euclidean distance for each point in the source point cloud, forming a point-to-point correspondence.

[0076] The second step is to solve for the transformation matrix. By minimizing the mean square distance error between all pairs of points, an optimal rigid body transformation matrix is ​​calculated. This matrix contains a 3×3 rotation matrix and a 3×1 translation vector.

[0077] This rigid body transformation is applied to the source point cloud, and then the algorithm repeats the above steps until the mean square error change between two consecutive iterations is less than the preset threshold of 0.001, or the number of iterations reaches the upper limit of 50.

[0078] The result of coarse registration is to provide a preliminary estimate of the workpiece pose, roughly aligning the current point cloud with the model point cloud.

[0079] The fine registration stage, based on the coarse registration, further utilizes the rich semantic information contained in the fused feature vector to improve the registration accuracy, especially when the workpiece is partially occluded or subject to noise interference.

[0080] Traditional iterative nearest-point algorithms rely solely on geometric distance, while this system introduces a soft constraint based on feature similarity.

[0081] The specific method involves incorporating feature similarity as an additional term in the registration loss function, optimizing it together with the geometric point-pair distance. The fine registration process aims to solve for a more accurate transformation matrix. The matrix consists of a rotation matrix. and a translation vector Composition. Its optimization objective function is defined as:

[0082] ;

[0083] in, It is the number of valid point pairs. These are points in the source point cloud. In the target point cloud The corresponding nearest point. and They are points and points Feature representation in the fused feature space.

[0084] It is a feature similarity function, and here we use cosine similarity to calculate it. Its value range is from 0 to 1. The larger the value, the more similar the features are.

[0085] It is a weighting factor that can be dynamically adjusted based on the distance or feature similarity between point pairs.

[0086] It is a balancing parameter used to adjust the relative importance of the geometric error term and the feature similarity term in the total loss, with a typical value of 0.5.

[0087] This optimization problem is solved using the Levenberg-Marquardt algorithm. By introducing feature similarity constraints, the fine registration process requires not only geometric alignment of the point clouds but also consistency in the semantic features of corresponding points. This enhances the robustness of registration and the accuracy of the final pose estimation under partial occlusion, noise, or interference from similar geometric structures.

[0088] The final output of the fine-registration is the three-dimensional translation vector and rotation matrix of the workpiece relative to the world coordinate system, which is the complete six-degree-of-freedom pose.

[0089] Please refer to the attached document. Figure 5 The dynamic identification and decision-making module is the intelligent decision-making center of this system. It integrates the output information of all preceding modules to make the final determination of the workpiece's identity and status.

[0090] This module contains two core sub-units: a confidence assessment unit and a decision arbitration unit.

[0091] The confidence assessment unit receives and processes two key reliability metrics in parallel.

[0092] The first type of metric is the two-dimensional recognition confidence score derived from the two-dimensional image feature extraction unit.

[0093] The score is derived from the output of the Softmax layer at the end of the deep convolutional neural network.

[0094] The Softmax layer transforms the raw scores output by the final fully connected layer of the network into a probability distribution vector. Each element of the vector corresponds to a possible artifact category, and its value represents the probability that the network believes the input image belongs to that category.

[0095] To obtain a more representative confidence score after calibration, this system uses a temperature scaling method to post-process the Softmax output, taking the maximum value in the probability vector as the confidence score for this two-dimensional recognition. Its value range is from 0 to 1.

[0096] The second type of metric is the 3D matching score from the 3D spatial reconstruction module.

[0097] This score is based on the final average matching error during the fine-registration phase. Calculated.

[0098] Average matching error It is the average Euclidean distance of all valid point pairs after the optimal transformation.

[0099] 3D matching score The formula maps the values ​​to the interval between 0 and 1 using an inverse proportional function.

[0100] ;

[0101] in It is a scaling factor, usually set to 10, used to map a typical error range to a suitable confidence interval.

[0102] The smaller the value, the higher the 3D matching degree. The closer it is to 1.

[0103] The decision arbitration unit receives the two-dimensional identification confidence score from the confidence assessment unit. 3D matching score .

[0104] The core mechanism of this unit is based on a dynamic decision threshold. Conduct intelligent arbitration.

[0105] Dynamic decision threshold It is not a fixed value, but rather an adaptive adjustment based on real-time environmental conditions and historical system performance.

[0106] Its benchmark value Set to 0.85.

[0107] The adjustment process considers two factors: ambient light intensity. Success rate of environmental history identification .

[0108] The system continuously monitors the light intensity of the working area through an independent ambient light sensor, measured in lux.

[0109] Meanwhile, the system maintains a fixed-length queue of 100 recognition task history records to calculate recent recognition success rates. Dynamic decision threshold The final calculation formula is:

[0110] ;

[0111] in, This is the illumination adjustment amount. When the real-time illumination intensity I is below 300 lux or above 1500 lux, ,otherwise . This is a historical performance adjustment amount, based on the calculated historical recognition success rate. When it is below 95%, ,otherwise .

[0112] The decision arbitration logic is as follows: The system first compares... and The maximum value in With dynamic decision threshold .if ≥ If the system directly adopts the recognition result corresponding to the information source that generated the maximum score as the final output, then the system will directly adopt the recognition result corresponding to the information source that generated the maximum score as the final output.

[0113] For example, if It is 0.92. It is 0.78, and It is 0.88, because The system will adopt the two-dimensional recognition results.

[0114] if and All below However, the absolute difference between the two is less than a preset tolerance range. This system is set If the value is 0.15, the decision-making arbitration unit will initiate a weighted fusion judgment mechanism.

[0115] The mechanism calculates a weighted average of the two scores based on their respective historical accuracy rates.

[0116] The system maintains an accuracy statistic for both 2D recognition and 3D matching. and These statistics are updated periodically based on historical validation data. The final weighted score...

[0117] ;

[0118] Then the system will With a fixed, low fusion threshold Comparison, It is usually set to 0.7.

[0119] like If the system outputs the recognition result favored by the weighted score, then the system outputs the recognition result that the weighted score tends to indicate. Or both ratings are lower than And the difference is ≥ If the system determines that the identification has failed, it will output a specific failure status code.

[0120] The system control and output module, serving as the interface between the system and the external world, bears the dual responsibility of internal timing coordination and external communication. This module embeds a real-time scheduler responsible for generating and distributing the aforementioned hardware trigger signals, ensuring that the data acquisition module, feature processing module, reconstruction module, and decision-making module work collaboratively according to a strict time sequence. The scheduler is built on a real-time operating system, with its task cycle consistent with the data acquisition frame rate, i.e., 33.3ms. Data transfer between all modules is achieved through shared memory and message queues, supplemented by a semaphore mechanism for synchronization, preventing data contention and ensuring real-time processing.

[0121] In terms of output, this module encapsulates the final decision result of the dynamic identification decision module into a data packet that conforms to the mainstream industrial Ethernet protocol.

[0122] This system preferably supports PROFINET and EtherCAT protocols. The output data packet has a fixed structure, which includes the following key fields: a 32-bit workpiece identification code to uniquely identify the identified workpiece type; a three-dimensional translation vector containing 3 floating-point numbers in mm; a rotation matrix represented by quaternions containing 4 floating-point numbers to describe the three-dimensional orientation of the workpiece; a 16-bit overall confidence score for this identification, which is the result of the final adoption of the confidence value by the decision arbitration unit or the quantized result of the calculated confidence value; and an 8-bit system status word to indicate identification success, failure, or system warning.

[0123] A cyclic redundancy check (CRC) code is appended to the end of the data packet to ensure the integrity of data transmission.

[0124] The data packet is output in real time to downstream actuators, such as robot controllers or programmable logic controllers, via a dedicated industrial network interface card, driving them to perform subsequent operations such as gripping, assembly, or sorting.

[0125] The entire system is deployed on an industrial-grade computing platform equipped with a multi-core CPU, a high-performance graphics processor, and ample memory. The system software adopts a layered architecture: the bottom layer handles sensor control and data acquisition, the middle layer implements the core algorithm modules, and the upper layer manages the system and handles external communication. Through the detailed implementation described above, this system achieves high-precision, high-reliability identification and accurate positioning of workpieces in complex industrial scenarios such as changing lighting, surface reflection interference, workpiece stacking, and partial occlusion.

[0126] This embodiment also provides an alternative implementation scheme for a workpiece visual recognition system in a smart factory, which features specific optimizations in sensor configuration, feature fusion strategy, and decision logic to adapt to application scenarios involving the recognition of high-speed moving workpieces. The overall system architecture remains unchanged, but the specific implementation of the multimodal data acquisition module differs.

[0127] In the multimodal data acquisition module, considering the high-speed movement of the workpiece on the conveyor belt, this embodiment employs a combination of a global shutter high-speed industrial area scan camera and a structured light 3D projector, replacing the combination of a line laser 3D scanner and a near-infrared depth sensor. The acquisition rate of the high-speed industrial area scan camera is increased to 120fps to capture clear 2D images with lower motion blur.

[0128] Structured light 3D projectors employ digital light processing technology to quickly reconstruct the 3D shape of a workpiece by projecting a series of coded light spot patterns onto the workpiece surface and simultaneously capturing the deformed patterns with a high-speed camera. This approach can reduce the single measurement time to less than 5ms, making it ideal for dynamic scenarios.

[0129] All sensors, including high-speed cameras and structured light projectors, are still synchronized through high-precision hardware triggering. The timing of the trigger signals is precisely calculated to ensure that data is acquired at a specific phase of the workpiece movement, reducing motion artifacts.

[0130] In the feature extraction and fusion module, considering the higher requirements for computational efficiency in high-speed applications, the structure of the feature alignment network has been simplified in this embodiment.

[0131] This feature alignment network is still a multilayer perceptron, but the number of layers is reduced to two fully connected layers. The number of nodes in the input layer corresponds to the concatenation dimension of the two-dimensional and three-dimensional feature vectors.

[0132] The number of nodes in the first fully connected layer is set to 512, and the number of nodes in the second fully connected layer is the final fusion feature dimension, which is set to 128.

[0133] The activation function employs a leak-corrected linear unit to prevent the gradient vanishing problem. This simplified network structure reduces computational cost and inference time while maintaining certain fusion performance, meeting the real-time requirements of high-speed processing.

[0134] The 3D spatial reconstruction module also adjusts its point cloud registration strategy. Since point clouds acquired by structured light may generate some noise due to motion, this embodiment introduces a global descriptor based on a fast point feature histogram for initial pose estimation in the coarse registration stage, in addition to using the iterative nearest-point algorithm, to improve the robustness and convergence speed of coarse registration. In the fine registration stage, the balance parameter λ in the optimization objective function is set to a small value, such as 0.2. This means that in high-speed scenes, the system relies slightly more on geometric consistency, while appropriately reducing the requirement for feature semantic consistency to cope with slight feature distortion that may be caused by motion.

[0135] In the confidence evaluation unit of the dynamic recognition decision module, for high-speed two-dimensional images, in addition to the Softmax output of the deep convolutional neural network, an auxiliary confidence index based on image sharpness is also introduced for the confidence score.

[0136] This metric is obtained by calculating the Laplacian variance of the image. When the variance value is below a certain threshold, the original two-dimensional recognition confidence score will be reduced to reflect the possible decrease in reliability due to motion blur.

[0137] In the decision-making arbitration unit, the adaptive adjustment strategy of the dynamic decision threshold adds consideration to the workpiece motion speed. The system obtains the real-time linear speed of the conveyor belt through an encoder. .

[0138] When speed When the speed exceeds the preset normal range, such as 5 m / min, the dynamic decision threshold is activated. The benchmark value It will be temporarily increased from 0.85 to 0.9 to impose stricter recognition standards and compensate for the possible decline in information quality caused by high-speed motion.

[0139] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0140] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A workpiece visual recognition system for smart factories, characterized in that, include: The multimodal data acquisition module is used to synchronously acquire multi-source sensing data of the workpiece. The multimodal data acquisition module is equipped with a high-resolution industrial area array camera, a line laser 3D scanner and a near-infrared depth sensor. The near-infrared depth sensor synchronously acquires the depth image information of the workpiece. All sensors achieve microsecond-level time synchronization through hardware trigger signals and are collected by a unified data interface. The feature extraction and fusion module is used to process the collected multi-source data and generate fused features. The feature extraction and fusion module includes a two-dimensional image feature extraction unit, a three-dimensional geometric feature extraction unit, and a multimodal feature fusion unit. The 3D space reconstruction module accurately reconstructs the 3D model of the workpiece in physical space and determines its pose based on the fused features and the original 3D data. The 3D space reconstruction module first uses the iterative nearest point algorithm to coarsely register the 3D point cloud of the current frame with the pre-stored standard workpiece model. Then, it combines the semantic information contained in the fused feature vector and uses a feature-based point cloud registration method to perform fine registration, thereby calculating the 3D translation vector and rotation matrix of the workpiece relative to the world coordinate system, i.e., the six-degree-of-freedom pose. The dynamic identification and decision-making module is used to make the final identity and status determination based on the output of the preceding module. The dynamic identification and decision-making module has a built-in confidence assessment unit and a decision arbitration unit. The system control and output module is used to coordinate the timing of various modules within the system and convert the final decision result of the dynamic identification decision module into an industrial bus protocol signal for output to the downstream actuator.

2. The workpiece visual recognition system for smart factories according to claim 1, characterized in that, The two-dimensional image feature extraction unit uses a deep convolutional neural network to process the color image to extract the texture, contour and color features of the workpiece; The three-dimensional geometric feature extraction unit preprocesses the three-dimensional point cloud data, including denoising, downsampling, and normal vector calculation, and extracts the surface curvature, edge lines, and key point features of the workpiece. The multimodal feature fusion unit receives the outputs from the two-dimensional and three-dimensional feature extraction units and maps the features of different modalities to a unified feature space through a feature alignment network for weighted concatenation to form a fused feature vector.

3. The workpiece visual recognition system for smart factories according to claim 1, characterized in that, The confidence evaluation unit receives in parallel a two-dimensional recognition confidence score from the two-dimensional image feature extraction unit and a three-dimensional matching score from the three-dimensional space reconstruction module. The two-dimensional recognition confidence score is obtained by calibrating the output of the Softmax layer of the deep convolutional neural network; The three-dimensional matching score is mapped from the average matching error after fine registration to the range of 0 to 1 via an inverse proportional function; The decision arbitration unit presets a dynamic decision threshold and adaptively adjusts it based on the ambient light sensor readings and historical recognition success rates. When the maximum value of the two-dimensional recognition confidence score and the three-dimensional matching score is higher than the dynamic decision threshold, the system adopts the recognition result corresponding to the higher score. When both scores are lower than the dynamic decision threshold and the difference between them is less than the preset tolerance range, the decision arbitration unit initiates a weighted fusion decision, weights the two scores according to their historical accuracy, and uses the weighted score as the final decision basis.

4. The workpiece visual recognition system for smart factories according to claim 1, characterized in that, The line laser 3D scanner and near-infrared depth sensor in the multimodal data acquisition module are orthogonally arranged in space.

5. A workpiece visual recognition system for a smart factory according to claim 2, characterized in that, The feature alignment network used in the feature extraction and fusion module is a multilayer perceptron with three fully connected layers.

6. A workpiece visual recognition system for a smart factory according to claim 1, characterized in that, The three-dimensional spatial reconstruction module introduces semantic information from the fused feature vector as a soft constraint for registration during the point cloud fine registration stage. Specifically, feature similarity is used as a term in the loss function and optimized together with the point-to-point distance.

7. A workpiece visual recognition system for a smart factory according to claim 6, characterized in that, The objective function of the loss function is defined as a weighted sum of the geometric error term and the feature similarity term, where the balancing parameter is used to adjust the relative importance of the two terms in the total loss.

8. A workpiece visual recognition system for a smart factory according to claim 1, characterized in that, The system control and output module supports mainstream industrial Ethernet protocols, and its output data packets include the workpiece's identification code, six-degree-of-freedom pose information, and the overall confidence score of this identification.

9. A workpiece visual recognition system for a smart factory according to claim 1, characterized in that, The three-dimensional geometric feature extraction unit uses a statistical outlier removal algorithm to denoise the original point cloud and a voxel grid downsampling method to reduce the point cloud density to 20% of the original density.

10. A workpiece visual recognition system for a smart factory according to claim 2, characterized in that, The feature alignment network in the multimodal feature fusion unit is trained with a large number of samples containing paired two-dimensional images and three-dimensional point clouds. Its learning objective is to minimize the Euclidean distance between different modal features from the same workpiece in a unified feature space after being mapped by the network.

Images