Visual guidance system and method for unstacking irregular plates

Abstract

The invention discloses a visual guidance system and method for unstacking irregular plate bodies, and the system comprises an image collection module which is used for obtaining the image data of the plate bodies through at least one camera; the data preprocessing module is used for preprocessing the acquired image data; the deep learning module comprises at least one convolutional neural network and is used for automatically extracting high-level features of the plate body from the preprocessed image data; the plate body detection and segmentation module is used for carrying out further semantic segmentation on the extracted features; the target identification and positioning module is used for identifying and tracking each individual target of the board body; the motion control module is used for controlling the unstacking robot to execute specific unstacking actions according to the position information provided by the target recognition and positioning module, the visual guidance system for unstacking the irregular plate bodies provides a more intelligent and automatic unstacking solution under the condition of irregular stacking, and the working efficiency is improved. By reducing manual intervention, the working efficiency is improved, and the safety risk is reduced.

Description

Technical Field

[0001] The invention relates to a visual guidance system and method for destacking irregular plates. Background Art

[0002] In modern warehousing, logistics and manufacturing, depalletizing has gradually become a key link. Especially when dealing with special products such as marine decorative panels, depalletizing is more challenging. Marine decorative panels usually have different sizes, shapes, materials and surface characteristics, so their stacking state is often irregular, which brings additional difficulties to traditional automated depalletizing systems.

[0003] Although the existing automated depalletizing system has made some progress in specific environments, the traditional method still has many technical bottlenecks and defects, which are mainly reflected in the following aspects: Marine decorative panels vary in shape, thickness and surface texture, and may be staggered, overlapped, and often at complex angles and positions when stacked. This is different from traditional neatly stacked panels, making it difficult for traditional automated systems to accurately grasp them.

[0004] Traditional vision-guided systems often rely on simple algorithms (such as edge detection or template matching), and often have difficulty accurately extracting high-level feature information when faced with marine decorative panels with complex surface features, color differences, and textures. This can result in low recognition accuracy and inability to accurately determine the position and posture of the panel, affecting subsequent destacking operations.

[0005] Even if the vision system can provide relatively accurate information about the position of the board, the traditional motion control system still faces the problem of accurate grasping when performing destacking operations, especially when facing complex arrangements of marine decorative panels. The robot may fail to operate or even damage the board or other equipment due to inaccurate path planning and grasping errors.

[0006] Marine decorative panels usually have special surface treatments, such as coatings or decorative surfaces, which make the panel surface very sensitive to collision and friction. Traditional robotic arms or gripping tools may damage the surface and affect the quality of the panel, especially in an environment with high precision requirements.

[0007] In complex environments, the real-time and dynamic adjustment capabilities of depalletizing operations are crucial. Traditional systems may not be able to respond to environmental changes in a timely manner, resulting in operational delays or errors. Summary of the invention

[0008] The purpose of the present invention is to provide a visual guidance system and method for destacking irregular panels, which can solve many problems in the traditional destacking method of marine decorative panels, especially in the case of irregular stacking, and provide a more intelligent and automated destacking solution. By reducing manual intervention, improving work efficiency and reducing safety risks, the system improves productivity while reducing overall operating costs.

[0009] The technical solution adopted by the present invention to solve its technical problem is: A visual guidance system for destacking irregular plates, comprising: An image acquisition module, used to obtain image data of the board through at least one camera, wherein the image data includes the shape, surface features, position and posture information of the board; The data preprocessing module is used to preprocess the acquired image data, including image denoising, enhancement, and normalization processing; A deep learning module, including at least one convolutional neural network, for automatically extracting high-level features of the board from the preprocessed image data, and identifying and locating the contour, boundary, and shape geometric features of the irregular board; The board detection and segmentation module is used to further segment the extracted features and identify the position, orientation, posture and specific arrangement of the board in the stack; The target recognition and positioning module is used to identify and track each individual target of the board, and output the precise position, rotation angle and destacking sequence information of the board in space; The motion control module controls the depalletizing robot to perform specific depalletizing actions based on the position information provided by the target recognition and positioning module, and completes the grabbing, handling and releasing operations of the board.

[0010] Another technical problem to be solved by the present invention is to provide a visual guidance method for destacking irregular plates, comprising the following steps: Acquire image data of the board through at least one camera, the image data including geometric shape, surface features and relative position information of the board; Perform pre-processing operations such as denoising, enhancement, and normalization on the collected image data; Use convolutional neural networks to automatically extract the geometric shape, texture features, and surface features of the board from the preprocessed images, and identify the boundaries and key features of irregular boards; Through semantic segmentation and instance segmentation algorithms, the position, outline and spatial relationship between the plates are identified; According to the position and posture information of the board, identify the spatial position, orientation and destacking order of each board; According to the information output by the target positioning and recognition module, the robot control system executes specific destacking actions to complete the grabbing and transportation of the board.

[0011] Preferably, the method for obtaining the image data of the plate body is: Select at least one industrial camera and place it at the position of the plate to take pictures, so as to capture the image of the entire plate or multiple plates; Use an industrial camera to capture the image data of the board in real time, and use the camera to continuously capture images from multiple perspectives; The image data captured by the camera includes the external geometry, boundaries, surface features, and relative position and posture information of the board.

[0012] Preferably, the method for preprocessing the collected image data for denoising, enhancement and normalization is: Use median filtering to remove image noise; Increase data diversity through rotation, translation, scaling, and contrast adjustment; Normalize the image pixel values ​​to [0, 1].

[0013] Preferably, a convolutional neural network is used to automatically extract the geometric shape, texture features and surface features of the plate from the preprocessed image, and a method for identifying the boundaries and key features of the irregular plate is as follows: Extracting local features and global features from the preprocessed image through multiple convolutional layers, wherein the convolutional neural network model includes multiple convolutional layers, pooling layers and fully connected layers; The convolutional neural network model is used to extract geometric features, texture features and surface features in the image, wherein the geometric features include the outline, angle and shape of the plate, the texture features include the smoothness and roughness of the surface, and the surface features include cracks and scratches; Detecting the boundary of the irregular board through the output of the network and identifying the key feature points of the board, wherein the boundary detection includes extracting the boundary box and key point positions of the board based on a convolutional neural network; The detected boundaries are smoothed and morphological operations are used to optimize the recognition results.

[0014] Preferably, the method of extracting local features and global features from the preprocessed image through multiple convolutional layers is: Extract local features through convolution operation. Each convolution layer contains multiple convolution kernels. By performing convolution operation with the local area of ​​the input image, multiple local feature maps are obtained. The convolution layer extracts low-level features, mid-level features and high-level features of the image layer by layer. The pooling layer reduces the spatial size of the image by selecting the maximum or average value in the local area while retaining important feature information; The fully connected layer realizes comprehensive analysis of global features by connecting all neurons to the neurons in the next layer; In the convolution layer, each convolution kernel performs a convolution operation with a local area of ​​the image to extract the features of the local area. By superimposing multiple convolution layers, the complex local features of the image are gradually extracted. Based on the convolutional layer and pooling layer, after multiple layers of feature extraction, the full connection layer is used to integrate the features of the entire image; Perform feature analysis based on the output of the network.

[0015] Preferably, the method for identifying the position, outline and spatial relationship between the plates by semantic segmentation and instance segmentation algorithms is as follows: Use semantic segmentation algorithm to classify each pixel in the input image, identify the object category to which each pixel belongs, and identify the board area in the image and its category information; Use instance segmentation algorithms to further distinguish different instances in the same category, generate mask information and bounding boxes for each board, and calibrate the specific position and outline of each board; Extracting the outline of the plate according to the mask information of the plate to obtain the precise shape of each plate; Calculate the spatial relationship between each board, including relative position, relative distance, relative angle, overlap detection and hierarchical relationship; Output recognition results, including the position and outline of each plate and the spatial relationship information between the plates.

[0016] Preferably, calculating the spatial relationship between the plates includes: Determine the relative position between the plates to determine the position of each plate relative to other plates; Calculate the relative distance between the plates to determine the proximity or distribution between the plates; Overlap detection between boards to determine whether there is an intersection area between the boards; The hierarchical relationship between the plates is inferred based on the position and shape information to determine whether the plate is above or below other plates.

[0017] Another technical problem to be solved by the present invention is to provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the visual guidance method for destacking irregular panels as described above is implemented.

[0018] Another technical problem to be solved by the present invention is to provide a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the visual guidance method for destacking irregular plates as described above.

[0019] The beneficial effects of the present invention are: Through the deep learning module and the target recognition and positioning module, the system can accurately identify the position, posture, rotation angle and arrangement of each board in the stack. This precise positioning information can help the depalletizing robot to grasp and carry with higher accuracy; the system can identify the board's shape, surface features, geometric features, posture and other multi-dimensional information, so that the depalletizing robot can adapt to different board shapes, sizes and arrangements, thereby improving the flexibility and efficiency of depalletizing.

[0020] Traditional destacking methods often rely on regularly arranged stacks, but this system uses advanced deep learning and visual guidance technology to identify and locate each board in irregular stacking situations, without being restricted by the arrangement method. Therefore, this solution is particularly suitable for complex or chaotic stacking environments; through semantic segmentation, the system can accurately segment overlapping or touching boards to prevent misgrabbing or misjudgment, thereby improving the reliability of destacking.

[0021] The system has a high degree of automation and can automatically identify, locate and plan the depalletizing sequence, thereby reducing human intervention, improving the level of automation, and reducing the occurrence of human errors. Based on the board information provided by the target recognition and positioning module, the motion control module can accurately control the depalletizing robot to perform the depalletizing action and automatically select the optimal path for grasping and transporting, thus optimizing the entire depalletizing process and reducing operation time.

[0022] The system can track the position changes of the panels in real time, ensuring that the robot can dynamically adjust its operating strategy during the depalletizing process to avoid depalletizing errors caused by position errors or changes. Through feedback from sensors and vision modules, the system can perform self-correction, optimize the robot's grasping strategy and action execution, and further improve the accuracy and stability of the depalletizing process. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 The present invention is a principle block diagram of a visual guidance system for destacking irregular plates. Specific implementation methods The principles and features of the present invention are described below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention. The present invention is described in more detail by way of example with reference to the accompanying drawings in the following paragraphs. The advantages and features of the present invention will become clearer according to the following description and claims. It should be noted that the drawings are all in a very simplified form and are not in precise proportions, and are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention.

[0025] In the description of the present invention, it should be understood that the terms "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", etc. indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific position, be constructed and operated in a specific position, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, it should be noted that, unless otherwise clearly specified and defined, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be directly connected, or indirectly connected through an intermediate medium, or it can be the internal communication of two elements.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0027] Example See also Figure 1 As shown, a visual guidance system for destacking irregular plates includes: An image acquisition module, used to obtain image data of the board through at least one camera, wherein the image data includes the shape, surface features, position and posture information of the board; The data preprocessing module is used to preprocess the acquired image data, including image denoising, enhancement, and normalization processing; A deep learning module, including at least one convolutional neural network, for automatically extracting high-level features of the board from the preprocessed image data, and identifying and locating the contour, boundary, and shape geometric features of the irregular board; The board detection and segmentation module is used to further segment the extracted features and identify the position, orientation, posture and specific arrangement of the board in the stack; The target recognition and positioning module is used to identify and track each individual target of the board, and output the precise position, rotation angle and destacking sequence information of the board in space; The motion control module controls the depalletizing robot to perform specific depalletizing actions based on the position information provided by the target recognition and positioning module, and completes the grabbing, handling and releasing operations of the board.

[0028] Through image acquisition, data preprocessing and deep learning modules, the system can automatically extract high-level features of the board, accurately identify and locate the position, posture and shape of the board. This means that the system can accurately identify the position and angle of each board in complex and irregular stacking scenes, eliminating manual intervention and deviation; using convolutional neural networks (CNN) for feature extraction and semantic segmentation, the system can still maintain a high recognition accuracy when facing a variety of board sizes, shapes and stacking methods.

[0029] Traditional depalletizing methods usually rely on manual positioning and disassembly, which is not only inefficient but may also cause damage or misoperation. The automated visual guidance system can significantly improve depalletizing efficiency and reduce human errors; the target recognition and positioning module can output the precise position, rotation angle and depalletizing sequence information of the board, ensuring the smooth depalletizing action, avoiding unnecessary interference and misoperation, and optimizing the depalletizing sequence and path planning.

[0030] The motion control module can accurately control the robot to grab, carry and release the board according to the identified target position and posture information. This accuracy ensures that even in high-density and complex stacking, the robot can still accurately perform tasks and avoid collisions and misgrasping; through deep learning and target recognition modules, the system can identify and track multiple board targets, realize multi-target parallel operation, and further improve destacking efficiency.

[0031] The system uses image enhancement, denoising and other technologies to enhance its adaptability to environmental changes (such as light changes, cluttered backgrounds, etc.), allowing the vision guidance system to operate stably in complex industrial environments; the system can be flexibly adjusted according to different board sizes, shapes and stacking conditions, and is suitable for different application scenarios. There is no need to redesign the hardware, only adjust the software to cope with new depalletizing tasks.

[0032] A visual guidance method for destacking irregular plates comprises the following steps: Acquire image data of the board through at least one camera, the image data including geometric shape, surface features and relative position information of the board; Perform pre-processing operations such as denoising, enhancement, and normalization on the collected image data; Use convolutional neural networks to automatically extract the geometric shape, texture features, and surface features of the board from the preprocessed images, and identify the boundaries and key features of irregular boards; Through semantic segmentation and instance segmentation algorithms, the position, outline and spatial relationship between the plates are identified; According to the position and posture information of the board, identify the spatial position, orientation and destacking order of each board; According to the information output by the target positioning and recognition module, the robot control system executes specific destacking actions to complete the grabbing and transportation of the board.

[0033] Through the automated visual guidance system, the entire depalletizing process does not require human intervention, reducing problems such as human errors and improper operation, and improving the efficiency and accuracy of depalletizing. The system can accurately identify and locate the position and posture of each board, reducing misgrasp and unnecessary pauses, and optimizing the overall efficiency of depalletizing, especially when faced with complex and irregular stacking.

[0034] This method can identify and process irregularly stacked boards, and does not require the stacking to have certain regularity or specifications. Regardless of how the boards are placed, the visual guidance system can identify the geometric shape, texture and spatial relationship of each board to ensure the adaptability of the system; through the denoising, enhancement and normalization of image data, the system can maintain stable performance under different lighting, background and environmental conditions, improving the robustness of visual guidance.

[0035] By using convolutional neural networks (CNN) to automatically extract the geometric shape, texture features, and surface features of the board in the image, the system can accurately identify the boundaries and key features of the board, ensuring that the precise position, orientation, and posture of each board are effectively identified; semantic segmentation and instance segmentation algorithms can not only distinguish the outline of the board, but also identify the spatial relationship between the boards. This precise segmentation capability enables the system to handle the destacking task of multiple boards and avoid misidentification or cross-interference.

[0036] Based on the relative position, orientation, stacking status and other information of the boards, the system can intelligently identify and determine the order of destacking each board. This means that the system can prioritize the removal of the easiest-to-grasp boards, avoid unnecessary interference and collisions, and optimize the destacking path; through precise target positioning information, the robot control system can plan the optimal grasping path and destacking action based on the identified board position and posture, improve work efficiency and avoid duplication of work.

[0037] The method for obtaining the image data of the plate is: Select at least one industrial camera and place it at the position of the plate to take pictures, so as to capture the image of the entire plate or multiple plates; Use an industrial camera to capture the image data of the board in real time, and use the camera to continuously capture images from multiple perspectives; The image data captured by the camera includes the external geometry, boundaries, surface features, and relative position and posture information of the board.

[0038] By photographing the external geometry and boundaries of the board, the size, outline and position of each board can be accurately obtained. This is crucial for high-precision automated destacking, assembly and handling tasks, and can avoid misjudgment and misplacement operations; image data can also extract surface features of the board, such as texture, color and markings, which helps to distinguish different boards in the stack or identify damaged areas, improving recognition accuracy.

[0039] By continuously taking images from multiple perspectives, we can fully understand the shape and position of the board, avoiding blind spots or information loss that may be caused by a single perspective. This is especially important for boards with irregular shapes, dense stacking, or large position changes; multi-perspective data can better understand the spatial position and posture of the board, helping robots or other automated systems to accurately locate the relative position of the board, thereby optimizing the destacking path and action execution.

[0040] Through real-time shooting with industrial cameras, the system can instantly obtain the latest image data of the board and provide quick feedback. This is crucial for fast response and automated operations in dynamic environments, ensuring that robots or automated equipment can always process the board in its current state; real-time performance enables the system to dynamically adapt in complex or changing environments and adjust operating strategies in a timely manner. For example, when the relative position or posture of the board changes, the visual system can instantly update data and replan actions.

[0041] Image data not only provides two-dimensional information about the board, but can also infer the three-dimensional spatial posture of the board through multiple perspectives. This is very helpful for tasks that require precise grasping and positioning (such as robot destacking and assembly, etc.), which can avoid error accumulation and ensure precise control; through the fusion of multiple perspectives, the system can obtain the relative position relationship between the boards, avoid repeated grasping or collision, and improve work efficiency and safety.

[0042] The method for preprocessing the collected image data for denoising, enhancement and normalization is as follows: Use median filtering to remove image noise; Increase data diversity through rotation, translation, scaling, and contrast adjustment; Normalize the image pixel values ​​to [0, 1].

[0043] Median filtering is a commonly used image denoising method that can effectively remove salt and pepper noise from images. Compared with other denoising methods, such as Gaussian filtering, median filtering can remove noise while retaining edge information, thus helping to improve image quality and avoid the impact of noise on subsequent image analysis or machine learning models; denoising can eliminate interference information, making subsequent processing and model training more accurate, reducing errors caused by noise, and improving system accuracy.

[0044] Through these data enhancement methods, the diversity of training data can be artificially increased, so that the model can learn features under different angles, positions, scales and lighting conditions, thereby improving the robustness and generalization ability of the model. This is especially important for training deep learning models, because the model can adapt to more complex scene changes and reduce overfitting; these enhancement operations help the model better adapt to various changes in the real environment, such as changes in shooting angles, differences in object size, changes in lighting conditions, etc., to ensure that the model can still perform well under different conditions.

[0045] Data augmentation increases data diversity and reduces the risk of overfitting by simulating various transformations during the training process, allowing more effective information to be utilized during the training process, thereby accelerating model convergence and improving training results. By performing diversified augmentation on image data, the model can better identify features when faced with unknown images, thereby improving recognition accuracy.

[0046] Normalization maps image pixel values ​​to a standard range (such as [0, 1]) to avoid numerical instability caused by differences in brightness, contrast, etc. between different images. Normalization can make image data more suitable for deep learning models and avoid gradient vanishing or gradient exploding problems during model training. Normalized image data is more suitable for input into neural networks for training because it can accelerate the convergence of the network and enable the model to reach the optimal state faster.

[0047] The method of using convolutional neural network to automatically extract the geometric shape, texture features and surface features of the plate from the preprocessed image and identify the boundaries and key features of the irregular plate is as follows: Extracting local features and global features from the preprocessed image through multiple convolutional layers, wherein the convolutional neural network model includes multiple convolutional layers, pooling layers and fully connected layers; The convolutional neural network model is used to extract geometric features, texture features and surface features in the image, wherein the geometric features include the outline, angle and shape of the plate, the texture features include the smoothness and roughness of the surface, and the surface features include cracks and scratches; Detecting the boundary of the irregular board through the output of the network and identifying the key feature points of the board, wherein the boundary detection includes extracting the boundary box and key point positions of the board based on a convolutional neural network; The detected boundaries are smoothed and morphological operations are used to optimize the recognition results.

[0048] Construct a network consisting of multiple convolutional layers, pooling layers, and fully connected layers to extract the geometric shape, texture features, and surface features of the image. The specific method is as follows: def build_model(input_shape): model = models.Sequential() # The first set of convolutional layers model.add(layers.Conv2D(32, (3, 3), activation='relu', padding='same', input_shape=input_shape)) model.add(layers.MaxPooling2D((2, 2))) # The second set of convolutional layers model.add(layers.Conv2D(64, (3, 3), activation='relu', padding='same')) model.add(layers.MaxPooling2D((2, 2))) # The third convolutional layer model.add(layers.Conv2D(128, (3, 3), activation='relu', padding='same')) model.add(layers.MaxPooling2D((2, 2))) # Flattening model.add(layers.Flatten()) # Fully connected layer model.add(layers.Dense(512, activation='relu')) model.add(layers.Dense(256, activation='relu')) # Output layer model.add(layers.Dense(4)) # Assume the output is four coordinates (x1, y1, x2, y2) representing the bounding box # Use mean squared error as loss function model.compile(optimizer='adam', loss='mean_squared_error') return model # Build the model input_shape = (128, 128, 3) # Assume the input image size is 128x128x3 model = build_model(input_shape) # View model architecture model.summary() When training the model, use the standard training process to pass image data and labels into the model.

[0049] # Train the model history = model.fit(X_train, y_train, epochs=20, batch_size=32,validation_data=(X_test, y_test)) After training is complete, the model is used to make predictions and extract bounding boxes and key point locations.

[0050] # Make predictions and get the bounding box or key point location output by the model predictions = model.predict(X_test) # Assume the model output is a bounding box (x1, y1, x2, y2) for i in range(len(predictions)): pred_bbox = predictions[i] # Map the predicted bounding box to the original image image = X_test[i] # Draw the bounding box plt.imshow(image) plt.gca().add_patch(plt.Rectangle((pred_bbox[0], pred_bbox[1]), pred_bbox[2]-pred_bbox[0], pred_bbox[3]-pred_bbox[1], linewidth=2, edgecolor='r', facecolor='none')) plt.show() To optimize the detected boundaries, morphological operations are applied to smooth the boundaries; def morphological_operations(image, kernel_size=(5, 5)): # Grayscale the image gray = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY) # Perform binarization (assuming there are clear object boundaries in the image) _, thresh = cv2.threshold(gray, 127, 255, cv2.THRESH_BINARY) # Use morphological operations to optimize boundaries kernel = np.ones(kernel_size, np.uint8) processed_image = cv2.morphologyEx(thresh, cv2.MORPH_CLOSE, kernel) return processed_image # Perform morphological processing on an image processed_image = morphological_operations(X_test[0]) # Display the processed image plt.imshow(processed_image, cmap='gray') plt.show() The detected boundaries are combined with the results of morphological optimization to improve the accuracy of boundary detection.

[0051] # Assume that the detected bounding box is pred_bbox, then perform morphological optimization for i in range(len(predictions)): pred_bbox = predictions[i] image = X_test[i] # Get the image after morphological optimization optimized_image = morphological_operations(image) # Draw the optimized bounding box plt.imshow(optimized_image, cmap='gray') plt.gca().add_patch(plt.Rectangle((pred_bbox[0], pred_bbox[1]), pred_bbox[2]-pred_bbox[0], pred_bbox[3]-pred_bbox[1], linewidth=2, edgecolor='r', facecolor='none')) plt.show() The method of extracting local features and global features from the preprocessed image through multiple convolutional layers is: Extract local features through convolution operation. Each convolution layer contains multiple convolution kernels. By performing convolution operation with the local area of ​​the input image, multiple local feature maps are obtained. The convolution layer extracts low-level features, mid-level features and high-level features of the image layer by layer. The pooling layer reduces the spatial size of the image by selecting the maximum or average value in the local area while retaining important feature information; The fully connected layer realizes comprehensive analysis of global features by connecting all neurons to the neurons in the next layer; In the convolution layer, each convolution kernel performs a convolution operation with a local area of ​​the image to extract the features of the local area. By superimposing multiple convolution layers, the complex local features of the image are gradually extracted. Based on the convolutional layer and pooling layer, after multiple layers of feature extraction, the full connection layer is used to integrate the features of the entire image; Perform feature analysis based on the output of the network.

[0052] Through the stacking of convolutional layers, the model can automatically extract local features from low-level to high-level from the image.

[0053] Low-level features: The first layer of convolution may extract basic elements such as edges and textures.

[0054] Mid-level features: As the number of layers increases, convolutional layers are able to combine and extract more complex features, such as corners, color changes, shapes, etc.

[0055] High-level features: Deeper layers may recognize more abstract objects or scene features.

[0056] This simple-to-complex feature extraction pipeline enables the model to very effectively capture complex and diverse patterns in images without the need for manual feature design.

[0057] The pooling layer reduces the spatial dimensions (such as width and height) of the image through maximum pooling or average pooling operations, thereby reducing the amount of computation while retaining important feature information. The advantages of pooling include: Reduced computation: Since the pooling layer reduces the size of the image, the computation of subsequent layers is reduced.

[0058] Increase the robustness of the model: The pooling operation is translation invariant, which means that even if the feature positions in the input image change slightly, the pooling layer can extract similar features, thereby improving the model's tolerance to small deformations or offsets.

[0059] Reduce the risk of overfitting: Pooling helps reduce the number of parameters of the model, thereby reducing the risk of overfitting.

[0060] Based on the convolutional and pooling layers, the features of different levels are extracted and combined layer by layer, which enables the model to: Automatically learn features: Through the back-propagation algorithm, the convolutional neural network will learn effective features based on the data without the need to manually design complex feature extraction algorithms.

[0061] Improved representation capability: Features from different layers can be combined in subsequent layers to form a global feature representation that is more meaningful for the target task (such as classification or detection).

[0062] Global feature integration in fully connected layers Based on the local features extracted by the convolutional layer and the pooling layer, the fully connected layer (FC) integrates and summarizes the local features by connecting each neuron with the neurons in the next layer to form a global understanding of the entire image. The functions of the fully connected layer include: Comprehensive analysis: The fully connected layer integrates the local features extracted by the convolutional layer and the pooling layer to help the model obtain global semantic information from the local features.

[0063] Information transmission: The neurons in each layer are connected to all the neurons in the next layer to form a high-dimensional feature representation, which enables the model to capture more complex and abstract image information.

[0064] Automatic feature learning, reducing manual intervention Convolutional neural networks can automatically learn features from data without relying on manual feature engineering. This makes the model: Strong adaptability: Features can be adaptively learned based on different input data without the need to manually specify and tune feature extraction methods.

[0065] Easy to expand and generalize: The model can be trained on a variety of tasks and datasets, and can be adapted to different application scenarios through fine-tuning.

[0066] Through semantic segmentation and instance segmentation algorithms, the method of identifying the position, outline and spatial relationship between the plates is as follows: Use semantic segmentation algorithm to classify each pixel in the input image, identify the object category to which each pixel belongs, and identify the board area in the image and its category information; Use instance segmentation algorithms to further distinguish different instances in the same category, generate mask information and bounding boxes for each board, and calibrate the specific position and outline of each board; Extracting the outline of the plate according to the mask information of the plate to obtain the precise shape of each plate; Calculate the spatial relationship between each board, including relative position, relative distance, relative angle, overlap detection and hierarchical relationship; Output recognition results, including the position and outline of each plate and the spatial relationship information between the plates.

[0067] Semantic segmentation can classify each pixel and accurately identify different object categories in the image, such as board areas. Each pixel in the image is assigned a category label, which allows the model to understand the distribution of objects in the image and extract relevant board areas; instance segmentation further solves the problem of distinguishing between objects of the same category, and can distinguish different instances (i.e., multiple independent parts of the same object) in complex scenes, thereby accurately calibrating the specific position and outline of each board.

[0068] The instance segmentation algorithm can not only identify the category of the board, but also generate mask information for each board to accurately depict the shape of the object. Through this mask information, the model can extract the precise outline of the board, rather than a simple bounding box or area, to obtain more detailed shape data; this process avoids errors that may occur in traditional methods, such as overlapping objects or complex geometric shapes, making the boundaries and positions of each board clearer.

[0069] Calculating the spatial relationships between the plates (such as relative position, distance, angle, overlap detection and hierarchical relationship) is a major advantage of this solution. These spatial relationships are crucial for further tasks such as relative positioning of objects, path planning, collision detection, etc. The accurate position and contour data obtained through instance segmentation can further calculate the geometric relationships between the plates, such as their relative position, angle, distance, etc., to help the model understand the relationship between different plates; the overlap detection function can help detect whether there is a collision or overlap between the plates, which is crucial for precise control and positioning.

[0070] Instance segmentation can distinguish multiple different objects in the same category and is suitable for scenes with multiple object instances. For example, in an image, there may be multiple boards whose shapes, positions, and layouts need to be accurately analyzed. Instance segmentation can avoid misjudging these boards as a single object and achieve accurate recognition and segmentation of multiple objects; for the management and operation of multiple boards in industrial automation or logistics scenarios, it is particularly important to accurately identify each board and understand the spatial relationship between them.

[0071] Calculating the spatial relationship between each plate includes: Determine the relative position between the plates to determine the position of each plate relative to other plates; Calculate the relative distance between the plates to determine the proximity or distribution between the plates; Overlap detection between boards to determine whether there is an intersection area between the boards; The hierarchical relationship between the plates is inferred based on the position and shape information to determine whether the plate is above or below other plates.

[0072] Relative position determination: By determining the positional relationship of each board relative to other boards (such as top, bottom, left, right, etc.), the relative layout between objects can be accurately understood. This is very important for scenarios such as automated sorting, transportation, and warehouse management to ensure the precise positioning of each object; positional relationship analysis helps optimize workflow and space planning. For example, in the manufacturing industry, determining the positional relationship of components on the production line helps improve production efficiency.

[0073] Relative distance calculation can determine the proximity or distribution between panels, and then help determine the compactness, dispersion, and potential space waste of objects in space. For example, in warehouse management, calculating the distance between panels can optimize the stacking strategy of items, reduce space waste, and increase storage density; in logistics and supply chain management, reasonable panel distribution can not only improve warehouse space utilization, but also help formulate more efficient handling strategies and loading methods.

[0074] Overlap detection is an important aspect of spatial relationship analysis, which can determine whether there is an intersection between plates. This is crucial for determining whether objects collide and need to be rearranged; in automated grasping, handling or stacking tasks, overlap detection can effectively avoid physical interference or collision between objects, ensuring that robots and automated equipment can perform tasks safely and efficiently.

[0075] Inferring hierarchical relationships based on position and shape information (for example, a plate is located above or below other plates) can help establish a hierarchical structure between plates and further optimize the spatial layout of objects. For example, in multi-layer storage systems, stacking scenes, or architectural design, judging the hierarchical relationship of plates can help arrange objects more reasonably; in industrial production and logistics, inferring hierarchical relationships helps optimize handling and stacking strategies to avoid unreasonable stacking methods that lead to unstable structures or waste of space.

[0076] This embodiment also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the visual guidance method for destacking irregular panels as described above when executing the program.

[0077] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the visual guidance method for destacking irregular plates as described above is implemented.

[0078] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM). Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above.

[0079] The above embodiments of the present invention are not intended to limit the protection scope of the present invention, and the implementation modes of the present invention are not limited thereto. All other modifications, replacements or changes made to the above structures of the present invention based on the above contents of the present invention, in accordance with common technical knowledge and customary means in the art, without departing from the above basic technical ideas of the present invention, should fall within the protection scope of the present invention.

Claims

1. A visual guidance system for destacking irregular panels, characterized in that: Included are: An image acquisition module, used to obtain image data of the board through at least one camera, wherein the image data includes the shape, surface features, position and posture information of the board; The data preprocessing module is used to preprocess the acquired image data, including image denoising, enhancement, and normalization processing; A deep learning module, including at least one convolutional neural network, for automatically extracting high-level features of the board from the preprocessed image data, and identifying and locating the contour, boundary, and shape geometric features of the irregular board; The board detection and segmentation module is used to further segment the extracted features and identify the position, orientation, posture and specific arrangement of the board in the stack; The target recognition and positioning module is used to identify and track each individual target of the board, and output the precise position, rotation angle and destacking sequence information of the board in space; The motion control module controls the depalletizing robot to perform specific depalletizing actions based on the position information provided by the target recognition and positioning module, and completes the grabbing, handling and releasing operations of the board.

2. A visual guidance method for destacking irregular panels, characterized in that: The following steps are involved: Acquire image data of the board through at least one camera, the image data including geometric shape, surface features and relative position information of the board; Perform pre-processing operations such as denoising, enhancement, and normalization on the collected image data; Use convolutional neural networks to automatically extract the geometric shape, texture features, and surface features of the board from the preprocessed images, and identify the boundaries and key features of irregular boards; Through semantic segmentation and instance segmentation algorithms, the position, outline and spatial relationship between the plates are identified; According to the position and posture information of the board, identify the spatial position, orientation and destacking order of each board; According to the information output by the target positioning and recognition module, the robot control system executes specific destacking actions to complete the grabbing and transportation of the board.

3. The visual guidance method for destacking irregular plates according to claim 2, characterized in that: The method for obtaining the image data of the plate is: Select at least one industrial camera and place it at the position of the plate to take pictures, so as to capture the image of the entire plate or multiple plates; Use an industrial camera to capture the image data of the board in real time, and use the camera to continuously capture images from multiple perspectives; The image data captured by the camera includes the external geometry, boundaries, surface features, and relative position and posture information of the board.

4. The visual guidance method for destacking irregular plates according to claim 2, characterized in that: The method for preprocessing the collected image data for denoising, enhancement and normalization is as follows: Use median filtering to remove image noise; Increase data diversity through rotation, translation, scaling, and contrast adjustment; Normalize image pixel values ​​to [0,1].

5. The visual guidance method for destacking irregular plates according to claim 4, characterized in that: The method of using convolutional neural network to automatically extract the geometric shape, texture features and surface features of the plate from the preprocessed image and identify the boundaries and key features of the irregular plate is as follows: Extracting local features and global features from the preprocessed image through multiple convolutional layers, wherein the convolutional neural network model includes multiple convolutional layers, pooling layers and fully connected layers; The convolutional neural network model is used to extract geometric features, texture features and surface features in the image, wherein the geometric features include the outline, angle and shape of the plate, the texture features include the smoothness and roughness of the surface, and the surface features include cracks and scratches; Detecting the boundary of the irregular board through the output of the network and identifying the key feature points of the board, wherein the boundary detection includes extracting the boundary box and key point positions of the board based on a convolutional neural network; The detected boundaries are smoothed and morphological operations are used to optimize the recognition results.

6. The visual guidance method for destacking irregular plates according to claim 5, characterized in that: The method of extracting local features and global features from the preprocessed image through multiple convolutional layers is: Extract local features through convolution operation. Each convolution layer contains multiple convolution kernels. By performing convolution operation with the local area of ​​the input image, multiple local feature maps are obtained. The convolution layer extracts low-level features, mid-level features and high-level features of the image layer by layer. The pooling layer reduces the spatial size of the image by selecting the maximum or average value in the local area while retaining important feature information; The fully connected layer realizes comprehensive analysis of global features by connecting all neurons to the neurons in the next layer; In the convolution layer, each convolution kernel performs a convolution operation with a local area of ​​the image to extract the features of the local area. By superimposing multiple convolution layers, the complex local features of the image are gradually extracted. Based on the convolutional layer and pooling layer, after multiple layers of feature extraction, the full connection layer is used to integrate the features of the entire image; Perform feature analysis based on the output of the network.

7. The visual guidance method for destacking irregular plates according to claim 2, characterized in that: Through semantic segmentation and instance segmentation algorithms, the method of identifying the position, outline and spatial relationship between the plates is as follows: Use semantic segmentation algorithm to classify each pixel in the input image, identify the object category to which each pixel belongs, and identify the board area in the image and its category information; Use instance segmentation algorithms to further distinguish different instances in the same category, generate mask information and bounding boxes for each board, and calibrate the specific position and outline of each board; Extracting the outline of the plate according to the mask information of the plate to obtain the precise shape of each plate; Calculate the spatial relationship between each board, including relative position, relative distance, relative angle, overlap detection and hierarchical relationship; Output recognition results, including the position and outline of each plate and the spatial relationship information between the plates.

8. The visual guidance method for destacking irregular plates according to claim 7, characterized in that: Calculating the spatial relationship between each plate includes: Determine the relative position between the plates to determine the position of each plate relative to other plates; Calculate the relative distance between the plates to determine the proximity or distribution between the plates; Overlap detection between boards to determine whether there is an intersection area between the boards; The hierarchical relationship between the plates is inferred based on the position and shape information to determine whether the plate is above or below other plates.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, the visual guidance method for destacking irregular plates as described in any one of claims 2 to 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a visually guided method for destacking irregular plates as described in any one of claims 2 to 8 is implemented.

Images