Intelligent material sorting method, device and equipment and storage medium

Abstract

The invention discloses an intelligent material sorting method, device and equipment and a storage medium, relates to the field of intelligent control, is applied to terminal equipment carrying a target material sorting system, and comprises the steps that target image data sent by target visual perception equipment corresponding to the target material sorting system is acquired; the target image data is image data obtained by acquiring original image data corresponding to the to-be-sorted material and preprocessing the original image data after the target visual perception equipment acquires a target acquisition signal; calling the target classification model by using a first target interface of the target material sorting system, so as to obtain a corresponding defect type detection result based on the target image data by using the target classification model; and the defect type detection result is written into a second PLC corresponding to material sorting equipment of the target material sorting system, so that the material sorting equipment is controlled to sort the to-be-sorted materials based on the defect type detection result. The material sorting efficiency and accuracy are improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent control, and in particular to an intelligent material sorting method, apparatus, equipment, and storage medium. Background Technology

[0002] With the continuous evolution of artificial intelligence technology, the integrated application of intelligent control and deep learning algorithms has become an important development direction in the industrial manufacturing field. Especially in industrial material management, sorting, and warehousing, such as in industrial scenarios like electronic product assembly lines, the rapid identification of minute defects directly affects product quality. Traditional manual operations suffer from low efficiency, poor accuracy, and high costs, making the introduction of efficient and precise intelligent solutions urgently needed. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide an intelligent material sorting method, apparatus, equipment, and storage medium, which can improve the efficiency and accuracy of material sorting. The specific solution is as follows:

[0004] In a first aspect, this application discloses an intelligent material sorting method, applied to a terminal device equipped with a target material sorting system, comprising: The system acquires target image data sent by the target visual sensing device corresponding to the target material sorting system; the target image data is the original image data corresponding to the material to be sorted acquired by the target visual sensing device after acquiring the target acquisition signal, and the image data is obtained after preprocessing the original image data; the target acquisition signal is the PLC signal triggered by the first PLC controller corresponding to the target placement platform after the material to be sorted is placed on the target placement platform; The target classification model is invoked using the first target interface of the target material sorting system, so as to obtain the defect type detection result corresponding to the material to be sorted based on the target image data using the target classification model; The defect type detection result is written into the second PLC controller corresponding to the material sorting equipment of the target material sorting system, so that the second PLC controller can control the material sorting equipment to sort the material to be sorted to the corresponding material storage area based on the defect type detection result.

[0005] Optionally, the preprocessing of the original image data includes: A target contour fitting algorithm is used to extract contours from the noise data in the original image data to obtain preliminary fitting results. The preliminary fitting result is optimized based on the target constraints to obtain the target fitting result corresponding to the original image data; the target constraints include radius constraints determined based on the size of the material to be sorted. The original image data is filtered based on the target fitting result to obtain filtered image data, and a target region segmentation algorithm is used to determine the candidate defect region corresponding to the filtered image data, so as to obtain the target image data corresponding to the material to be sorted based on the candidate defect region and the filtered image data.

[0006] Optionally, before determining the candidate defect region corresponding to the filtered image data using the target region segmentation algorithm, the method further includes: Determine whether there are noise points in the filtered image data, and determine whether the current window size of the filtering window has reached the maximum window limit; If there are noise points in the filtered image data and the filtering window has not reached the maximum window limit, the filtering window is adjusted based on a dynamic window adjustment strategy to obtain the adjusted filtering window. The filtered image data is filtered based on the adjusted filtering window and the target fitting result to obtain new filtered image data, and then the process jumps to the step of determining whether there are noise points in the filtered image data.

[0007] Optionally, before calling the target classification model using the first target interface of the target material sorting system, the method further includes: Obtain the target material classification requirements; Based on the target material classification requirements, a target classification model is determined from the target model pool; The target model pool includes a first classification model based on a lightweight network architecture, a second classification model based on a residual network architecture, and a third classification model based on a Transformer architecture.

[0008] Optionally, the step of using the target classification model to obtain the defect type detection result corresponding to the material to be sorted based on the target image data includes: If the target classification model is the first classification model based on a lightweight network architecture, then the model channel weights of the target classification model are determined based on a local cross-channel interaction strategy, so as to determine the target gradient information based on the model channel weights; The target classification model is used to obtain the defect type detection result corresponding to the material to be sorted based on the target image data. When the acquisition time of the defect type detection result is greater than the preset inference time threshold, the target channel pruning strategy is used to adjust the model channel weight of the target classification model based on the target gradient information to obtain the updated target classification model.

[0009] Optionally, the step of using the target classification model to obtain the defect type detection result corresponding to the material to be sorted based on the target image data includes: The target classification model is used to determine the target geometric features and target grayscale features corresponding to the candidate defect regions based on the target image data; Based on the target geometric features and the target grayscale features, the defect type corresponding to the candidate defect region is determined, and the defect type detection result corresponding to the material to be sorted is obtained based on the defect type corresponding to each candidate defect region of the target image data.

[0010] Optionally, the intelligent material sorting method further includes: The video data collected by the target visual perception device is transmitted in real time to the front-end interface corresponding to the target material sorting system, and the material sorting instructions are obtained through the front-end interface to sort the materials to be sorted on the target platform based on the material sorting instructions. The material sorting instructions include sorting start instructions, sorting stop instructions, and sorting mode adjustment instructions.

[0011] Secondly, this application discloses an intelligent material sorting device, applied to a terminal device equipped with a target material sorting system, comprising: The image data acquisition module is used to acquire target image data sent by the target visual sensing device corresponding to the target material sorting system; the target image data is the original image data corresponding to the material to be sorted acquired by the target visual sensing device after acquiring the target acquisition signal, and the image data is obtained after preprocessing the original image data; the target acquisition signal is the PLC signal triggered by the first PLC controller corresponding to the target platform after the material to be sorted is placed on the target platform; The defect type detection module is used to call the target classification model through the first target interface of the target material sorting system, so as to use the target classification model to obtain the defect type detection result corresponding to the material to be sorted based on the target image data; The first material sorting module is used to write the defect type detection result into the second PLC controller corresponding to the material sorting equipment of the target material sorting system, so as to use the second PLC controller to control the material sorting equipment to sort the material to be sorted to the corresponding material storage area based on the defect type detection result.

[0012] Thirdly, this application discloses an electronic device, including: Memory, used to store computer programs; A processor is used to execute the computer program to implement the aforementioned intelligent material sorting method.

[0013] Fourthly, this application discloses a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned intelligent material sorting method.

[0014] In this application, during intelligent material sorting, a terminal device equipped with a target material sorting system acquires target image data sent by a target visual sensing device corresponding to the target material sorting system. The target image data is the original image data of the material to be sorted acquired by the target visual sensing device after acquiring the target acquisition signal, and the image data is obtained after preprocessing the original image data. The target acquisition signal is a PLC signal triggered by the first PLC controller corresponding to the target placement platform after the material to be sorted is placed on the target placement platform. The target classification model is called using the first target interface of the target material sorting system to obtain the defect type detection result of the material to be sorted based on the target image data. The defect type detection result is written into the second PLC controller corresponding to the material sorting device of the target material sorting system to control the material sorting device to sort the material to be sorted to the corresponding material storage area based on the defect type detection result. As can be seen, in this application, once the material to be sorted is placed on the target shelf, the image acquisition and preprocessing actions of the target visual perception device are automatically triggered. After the target material sorting system obtains the target image data corresponding to the material to be sorted, it can call the target classification model and use the target classification model to process the target image data to determine the defect type of the material to be sorted. Finally, the defect type detection result corresponding to the material to be sorted is written to the second PLC controller corresponding to the material sorting equipment, thereby using the material sorting equipment to sort the material to be sorted to the corresponding material sorting area, completing the entire intelligent material sorting process and improving the efficiency and accuracy of material sorting. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort. Figure 1 This is a flowchart of an intelligent material sorting method disclosed in this application; Figure 2 This is a schematic diagram of the architecture of a target material sorting system disclosed in this application; Figure 3This is a schematic diagram of a specific Profinet network architecture disclosed in this application; Figure 4 This is a schematic diagram of a specific material sorting process disclosed in this application; Figure 5 This is a schematic diagram of a specific target image data processing result disclosed in this application, wherein... Figure 5 (a) is a schematic diagram of a scratch defect treatment. Figure 5 (b) is a schematic diagram of a hole defect treatment method. Figure 5 (c) is a schematic diagram of a stain defect treatment; Figure 6 This application discloses a specific real-time camera feedback image; Figure 7 This is a schematic diagram of a specific container architecture disclosed in this application; Figure 8 This is a sample image of a specific front-end monitoring webpage disclosed in this application; Figure 9 This is a schematic diagram of the structure of an intelligent material sorting device disclosed in this application; Figure 10 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] With the continuous evolution of artificial intelligence technology, the integrated application of intelligent control and deep learning algorithms has become an important development direction in the industrial manufacturing field. Especially in industrial material management, sorting, and warehousing, such as in electronic product assembly lines, the rapid identification of minute defects directly affects product quality. Traditional manual operations suffer from low efficiency, poor accuracy, and high costs, necessitating the introduction of efficient and precise intelligent solutions. To address these technical problems, this application discloses an intelligent material sorting method that can improve material sorting efficiency and accuracy.

[0018] See Figure 1 As shown, this embodiment of the invention discloses an intelligent material sorting method, applied to a terminal device equipped with a target material sorting system, including:

[0019] Step S11: Obtain target image data sent by the target visual sensing device corresponding to the target material sorting system; the target image data is the image data obtained by the target visual sensing device after acquiring the target acquisition signal, acquiring the original image data corresponding to the material to be sorted, and preprocessing the original image data; the target acquisition signal is the PLC signal triggered by the first PLC controller corresponding to the target placement platform after the material to be sorted is placed on the target placement platform.

[0020] In this embodiment, as Figure 2 The diagram shows the architecture of a target material sorting system. This system is an intelligent control system integrating deep learning visual recognition, three-axis servo control, automated material sorting, and warehouse management. The overall system architecture is divided into three layers: the perception layer, the decision layer, and the execution layer. The perception layer consists of industrial-grade visual inspection cameras and other visual perception devices, along with a placement platform. It is responsible for acquiring material images and transmitting them to an edge computing platform for processing. The decision layer, centered on edge computing devices, deploys optimized deep learning algorithm models and uses Python to write visual logic and classification strategies to achieve intelligent material identification and task decision-making. The system integrates the Podman container platform for model deployment, supporting TCP (Transmission Control Protocol) communication with PLCs (Programmable Logic Controllers) and achieving control interaction through signal synchronization of command streams. The execution layer consists of a three-axis servo gripping system and a PLC control platform. It is responsible for controlling the X / Y / Z three-axis motion and electromagnetic chuck adsorption actions based on visual inspection results, accurately placing materials into the corresponding warehouse shelf locations. This system uses the PROFINET bus protocol for real-time communication and, in conjunction with an HMI interface, enables task status monitoring and manual debugging. In one specific implementation, the target material sorting system includes electrical equipment such as the Phoenix edge controller (EPC 1502), IL PN BK-PAC, bus couplers, inline digital input / output modules, industrial switches, UNO2-PS / 24DC switching power supplies, ESTUN servo controllers (ED3L PN), and industrial cameras, all connected via Profinet communication. A detailed Profinet network architecture diagram is shown below. Figure 3 As shown in the diagram, the material sorting process is as follows: Figure 4 As shown. The system can exchange RscVariant data through PyPlcnextRsc's GDS (Global Data Service), and use the lightweight Python web framework Flask to create web applications to achieve low-latency rendering in the browser.

[0021] In this embodiment, after the material to be sorted is placed on the target platform of the target material sorting system, the first PLC controller corresponding to the target platform automatically triggers the PLC signal and transmits the PLC signal as a target acquisition signal to the target visual sensing device (such as an industrial camera) of the target material sorting system. The target visual sensing device then acquires an image of the material to be sorted and performs preliminary preprocessing on the acquired raw image data to obtain the corresponding target image data. Specifically, the preprocessing of the raw image data may include: extracting the contours of noisy data in the raw image data using a target contour fitting algorithm to obtain a preliminary fitting result; optimizing the preliminary fitting result based on target constraints to obtain the target fitting result corresponding to the raw image data; the target constraints include a radius constraint determined based on the size of the material to be sorted; filtering the raw image data based on the target fitting result to obtain filtered image data; and using a target region segmentation algorithm to determine the candidate defect regions corresponding to the filtered image data, thereby obtaining the target image data corresponding to the material to be sorted based on the candidate defect regions and the filtered image data. It is understood that, in this embodiment, before determining the candidate defect region corresponding to the filtered image data using the target region segmentation algorithm, the method may further include: determining whether there are noise points in the filtered image data, and determining whether the current window size of the filtering window has reached the maximum window limit; if there are noise points in the filtered image data, and the filtering window has not reached the maximum window limit, then the filtering window is adjusted based on the dynamic window adjustment strategy to obtain the adjusted filtering window; the filtered image data is filtered based on the adjusted filtering window and the target fitting result to obtain new filtered image data, and then the process jumps to the step of determining whether there are noise points in the filtered image data.

[0022] In one specific implementation, OpenCV can be used to preprocess the raw image data acquired by the industrial camera to improve the accuracy of subsequent detection algorithms. Specifically, for outliers in material edge detection, the random sampling consensus algorithm achieves noise-robust contour extraction (i.e., sub-pixel contour fitting) through iterative model fitting and interior point screening. Its core process is as follows: 1. Three-stage iterative optimization: In 500 random samplings, select 3 non-collinear points each time to calculate the center of the circle (algorithm complexity O(n)); 2. Adaptive tolerance control: Set a tolerance threshold of 2.5 pixels to filter inlier points. When the proportion of inlier points is >85%, trigger least squares fine-fit fitting. 3. Physical constraint verification: Limit the radius range to 160-200 pixels (corresponding to the size range of the material to be sorted is 50-80mm) to exclude abnormal fitting results.

[0023] Through the above preprocessing steps, after initial fitting, cubic spline interpolation is used to perform sub-pixel-level gradient calculations on the edge transition region, improving the contour positioning accuracy to the 0.1 pixel level, thus achieving sub-pixel accuracy improvement. Simultaneously, by mapping the material's physical dimensions (50-80mm) to the pixel range in the image space, the radius search interval is limited, significantly reducing invalid calculations.

[0024] In this embodiment, after extracting the sub-pixel contours of the original image of the materials to be sorted, a double-layered annular region of interest (ROI) can be constructed based on the center and radius of the circle accurately fitted by RANSAC. This is a circular mask with an inward shrinkage of 5 pixels, which can eliminate 98% of the background noise and achieve physical isolation of background interference signals. Specifically, to cope with the mixed interference of impulse noise and Gaussian noise, a dynamic window adjustment strategy can be adopted: 1. Initial filtering: Calculate the median of the neighborhood using a 3×3 window. If the current pixel value is an extreme value (exceeding the median ± 2 times the standard deviation), it is determined to be a noise point. 2. Window expansion: If the noise points are not effectively corrected, gradually expand the window to 7×7 until a valid median is found or the maximum window limit is reached.

[0025] Furthermore, while filtering out high-density noise (such as 15% salt-and-pepper noise), linear scratch features with a width of not less than 0.1 mm are fully preserved. Simultaneously, a dynamic window mechanism avoids edge blurring effects caused by large-size filter kernels. For the morphological discontinuities of minute defects (such as cracks and holes), multi-scale composite operations are designed. Based on the Otsu algorithm, dynamic threshold segmentation is performed on defects within a circle to eliminate the influence of uneven illumination and extract candidate defect regions. The area error rate of the defect region is controlled within 5%, significantly improving the accuracy of subsequent quantitative analysis. Complete connectivity repair can be achieved for cracks with a width greater than 0.05 mm. Based on the above preprocessing, this embodiment constructs a complete noise-reducing enhancement processing system.

[0026] Step S12: Use the first target interface of the target material sorting system to call the target classification model, so as to use the target classification model to obtain the defect type detection result corresponding to the material to be sorted based on the target image data.

[0027] In this embodiment, before calling the target classification model using the first target interface of the target material sorting system, the target material classification requirements can be obtained; based on the target material classification requirements, the target classification model is determined from the target model pool; wherein, the target model pool includes a first classification model based on a lightweight network architecture, a second classification model built on a residual network architecture, and a third classification model based on a Transformer architecture. This means that the target material sorting system in this embodiment supports multiple mainstream image classification network structures.

[0028] In one specific implementation, the system framework encapsulates modules such as model initialization, weight loading, and category adaptation, and can quickly switch between the following model architectures as needed: 1. Lightweight networks: MobileNetV, MobileNetV2, MobileNetV3; 2. Medium-sized residual networks: ResNet18, ResNet34; 3. Deep residual networks: ResNet50, ResNet101, ResNet152; 4. Classic VGG series: VGG11, VGG13, VGG16 (including *_bn batch normalized versions); 5. Visual Transformer Architecture; 6. ViT series: ViT-B / 16 (Vision Transformer); 7. Swin Transformer series: Swin-Tiny, Swin-Small, Swin-Base; All the models mentioned above are implemented in PyTorch and integrated into the `get_model_from_name` model factory, supporting a unified API call for easy and rapid testing of model performance differences or transfer learning. The Swin Transformer introduces a local window self-attention mechanism for higher accuracy, while the ViT-B / 16's global attention mechanism adapts to large field-of-view modes. This strategy not only ensures system stability and real-time performance but also provides interfaces for later model replacement and adaptation to complex scenes, exhibiting good engineering scalability and maintainability.

[0029] For example, to meet the multiple requirements of computational efficiency, accuracy, and inference speed in edge deployment environments, the lightweight convolutional neural network MobileNetV3 can be used as the main model architecture for object classification, specifically for object type identification or defect classification in images captured by industrial cameras. MobileNetV3 is a further optimized network architecture based on MobileNetV1 and MobileNetV2, aiming to achieve higher efficiency and better performance on resource-constrained platforms such as mobile devices by combining automated search techniques and network structure improvements. Compared to its predecessor, MobileNetV3 improves accuracy while maintaining high efficiency. MobileNetV3 has a simple structure, small parameter count, and high computational efficiency, making it very suitable for running without GPUs, while also achieving good classification results on medium-sized datasets.

[0030] In this embodiment, the target classification model built on MobileNetV3 includes an initial convolutional layer (InitialConv), improved depthwise separable convolutional blocks, an efficient channel attention (ECA) module, a hard swish (h-swish) activation function, optimized final stages after pooling, and a classification layer.

[0031] In one specific implementation, the improved depthwise separable convolutional block uses an inverted residual structure, i.e., first expanding the number of channels, performing depthwise convolution, and then compressing the number of channels using 1x1 convolution; the ECA module implements an efficient channel attention mechanism through one-dimensional convolution, avoiding the dimensionality reduction and information loss problems caused by the fully connected layers in the SE module, thus more effectively capturing cross-channel interaction information. This enhances the model's ability to focus on key channel features with fewer parameters and less computation, improving the discriminative power of feature representation and enhancing the model's expressive power, thereby achieving higher accuracy in material classification tasks; the h-swish activation function is used instead of ReLU or Swish in the later layers of the network. This is a smooth approximation of ReLU, which is more computationally efficient and easier to deploy on mobile devices. The early layers of the network still use ReLU, Pointwise... The convolution uses 1×1 convolutions to linearly combine multiple channels to construct new feature maps. Before the classification head, the target classification model optimizes the last few layers, removing the computationally intensive layers before the 1×1 convolutional layer in the last inverted residual block of MobileNetV2, and efficiently compressing high-dimensional features into a one-dimensional vector through global average pooling, which is then connected to the final classification layer. The classification layer contains a global average pooling layer that pools the feature maps into a one-dimensional vector, which is then connected to one or more fully connected layers to output the final classification result (in this example, four types of defects are classified, including chipped corners, stains, holes, and scratches). Finally, the softmax function is used to generate the probability distribution. MobileNetV3 also optimizes the classification head to improve efficiency. The specific network structure configuration is shown in Table 1 below, and the model structure and corresponding parameter quantities are shown in Table 2 below. Here, DWConv represents Depthwise convolution, and PWConv represents Pointwise convolution.

[0032] Table 1 Network Structure Configuration

[0033]

[0034] Table 2 Model Structure and Corresponding Parameters

[0035]

[0036] In this embodiment, to adapt to specific tasks and deployment environments, the following optimizations were also made during the model design and inference process: 1. Input image resolution cropping and scaling: The input image is uniformly scaled to 224×224 using the letterbox_image method to ensure that the aspect ratio remains unchanged and improve the stability of the model; 2. Inference process without GPU dependency: The system automatically determines torch.cuda.is_available(), and can still run stably on platforms without GPU; 3. Decoupled design of interface layer: The model structure, category definition, and input processing flow are all encapsulated in the Classification class, which makes it easy to replace with a better structure such as MobileNetV2 or EfficientNet-lite in the future; 4. Containerized deployment: The model and inference system are encapsulated in Podman containers, improving deployment consistency and ease of operation and maintenance; 5. Clear post-processing mapping: The category index output by the model corresponds one-to-one with the specific defect, simplifying the logic judgment burden on the PLC side.

[0037] In this embodiment, a target classification model is used to obtain the defect type detection results corresponding to the materials to be sorted based on target image data. This includes: using the target classification model to determine the target geometric features and target grayscale features corresponding to candidate defect regions based on target image data; determining the defect type corresponding to the candidate defect regions based on the target geometric features and target grayscale features; and obtaining the defect type detection results corresponding to the materials to be sorted based on the defect types corresponding to each candidate defect region in the target image data. In other words, by combining the geometric attributes and spatial distribution characteristics of defects and integrating grayscale feature analysis, intelligent identification and precise positioning of defects are achieved.

[0038] In one specific implementation, the target classification model first extracts geometric features and establishes a multi-screening mechanism based on area thresholds and edge distances. Shape complexity, i.e., compactness (e.g., scratches > 5, holes < 3), is quantified by calculating the ratio of the square of the perimeter to the area. Polar coordinate positioning is used to calculate the polar radius (distance) and polar angle (azimuth) of the defect's centroid relative to the center of the circle, thus determining the defect location. Simultaneously, grayscale feature analysis is performed to statistically analyze the proportion of black pixels within the defect area (e.g., stains > 80%). Based on the extracted features, the model makes intelligent classification decisions. False defects less than 15 pixels from the edge are eliminated through spatial constraints. Then, by combining multiple thresholds such as compactness, grayscale distribution, and edge distance, the model can distinguish the type of material and various defects (including scratches, missing corners, holes, stains, and other surface defects). The defect type detection results serve as a key input to the PLC control logic, guiding the three-axis gripping system to perform correct sorting operations. For standardized analysis, a rotation alignment technique is used, calculating the centroid polar angle and using affine transformation to rotate the defect to a standard angle. The processing results are as follows: Figure 5 As shown, where Figure 5 (a) is a schematic diagram of a scratch defect treatment. Figure 5 (b) is a schematic diagram of a hole defect treatment method. Figure 5 (c) is a schematic diagram of a stain defect processing method. This embodiment integrates multi-dimensional features such as compactness, polar coordinate position, and grayscale ratio to quantitatively describe defects, and achieves accurate classification and discrimination based on multi-threshold fusion. Rotation alignment is used to improve discrimination consistency, realizing multi-dimensional intelligent defect quantification and classification. Through mechanism innovation and algorithm optimization, a preprocessing enhancement system for complex industrial environments is constructed, successfully solving the problems of contour distortion, noise interference, and misjudgment in the detection of minute defects. Practical application shows that the system has a recall rate of over 94% for defects at the 0.1mm level and possesses good illumination robustness and real-time performance, providing reliable technical support for intelligent manufacturing quality control.

[0039] In one specific implementation, if the target classification model is a first classification model based on a lightweight network architecture, the model channel weights of the target classification model are determined based on a local cross-channel interaction strategy, and the target gradient information is determined based on the model channel weights. The target classification model is then used to obtain the defect type detection results corresponding to the materials to be sorted based on the target image data. When the acquisition time of the defect type detection results exceeds a preset inference time threshold, the model channel weights of the target classification model are adjusted based on the target gradient information using a target channel pruning strategy to obtain an updated target classification model. In other words, in this embodiment, a gradient-sensitive channel pruning strategy can be integrated. After the target classification model is trained, the model channels are further pruned using gradient information. Gradient-sensitive pruning can identify and remove redundant channels that contribute little to the output of the target classification model, thereby significantly compressing the model size and reducing computational load without significantly sacrificing model performance. This post-processing optimization step makes the improved model more compact and efficient, particularly suitable for deployment on embedded systems or mobile devices with limited computing power and storage space, achieving model miniaturization and accelerated inference. Actual tests show that, when the target classification model is deployed on a platform (without GPU) using only CPU inference, the inference time for a single image is approximately 170~190 ms, which fully meets the requirements of the actual control system for real-time performance (inference time less than 1s) and stability.

[0040] To adapt to the dynamic needs of industrial scenarios, this embodiment introduces a dual-dimensional adjustable mechanism. This dual-dimensional adjustable mechanism includes an adjustment mechanism based on width multiplier and an adjustment mechanism based on resolution multiplier. The adjustment mechanism based on width multiplier (a) proportionally reduces the number of channels in each layer; when a=0.75, the number of model parameters decreases from 4.2M to 2.8M, improving computational efficiency by 40%. The adjustment mechanism based on resolution multiplier (b) dynamically adjusts the input image size (e.g., reducing it from 224×224 to 192×192), maintaining feature resolution through bilinear interpolation, reducing computation by 30%. At the model training level, the ReLU6 activation function (output limited to the [0,6] interval) is used to enhance the numerical stability of low-precision calculations, and FocalLoss is integrated to address the imbalance problem of material defect categories. To address illumination fluctuations, this embodiment provides an adaptive histogram equalization algorithm to compensate for illumination differences. Testing shows that the algorithm in this embodiment can accurately sort materials, completing the entire process in approximately 2 minutes.

[0041] Step S13: Write the defect type detection result into the second PLC controller corresponding to the material sorting equipment of the target material sorting system, so as to use the second PLC controller to control the material sorting equipment to sort the material to be sorted to the corresponding material storage area based on the defect type detection result.

[0042] In this embodiment, after obtaining the defect type detection result corresponding to the material to be sorted, the defect type detection result can be written into the second PLC controller corresponding to the material sorting equipment of the target material sorting system. Then, the second PLC controller uses the defect type detection result to control the material sorting equipment through a three-axis servo system, thereby sorting the material to be sorted to the corresponding material storage area. When using the three-axis servo system to sort the material to be sorted, methods such as dynamic programming and AI search can be used to optimize the task sequence. Combined with visual results, the action queue can be dynamically adjusted to achieve the optimal execution path of the task, improve response efficiency, and reduce energy consumption.

[0043] In this embodiment, the video data collected by the target visual perception device can also be transmitted in real time to the front-end interface corresponding to the target material sorting system, and material sorting instructions can be obtained through the front-end interface to sort the materials to be sorted on the target platform based on the material sorting instructions; wherein, the material sorting instructions include sorting start instructions, sorting stop instructions and sorting mode adjustment instructions.

[0044] In this embodiment, to verify the stability and performance of the target material sorting system in practical application scenarios, the following test experiments were designed and executed, focusing on key indicators such as visual recognition accuracy, sorting efficiency, environmental adaptability, and system robustness:

[0045] 1. Data Source and Testing Environment: The test data is based on standard materials and a self-constructed simulated scenario dataset. The standard materials consist of 12 iron sheets, covering four types of defects: scratches, stains, missing corners, and holes, with 3 samples of each defect type. Based on these standard materials, 100 sets of test data were generated, covering various interference scenarios. By adjusting the mixed lighting of LED ring light source (300~800Lux) and natural light, the reflection, shadow, or low-light environment that may occur in actual applications was simulated. The material rotation angle (0°~360°) and surface cleanliness (artificially added simulated stains) were randomly adjusted to perturb the material's state.

[0046] 2. Testing process and test results: The core test scenarios include four categories: standard environment, dynamic interference environment, random material disturbance, and continuous load test.

[0047] (1) Standard environment test: Under standard uniform lighting conditions, with no material rotation and no obstruction, the system completed the sorting of 12 materials in an average of 2 minutes with 100% accuracy. The three-axis gripping system has high path planning efficiency, with a single gripping-placement cycle taking 8~10 seconds, of which visual recognition and positioning account for about 20%, servo motion accounts for 65%, and suction cup adsorption / release accounts for 15%.

[0048] (2) Dynamic interference environment test: This mainly includes light source change test, in which the light intensity is randomly switched during the sorting process. The test results show that the missed detection rate of stain defects under low light conditions increased to 3.8%, and the overall task time increased to 2 minutes and 10 seconds.

[0049] (3) Random material disturbance test: The material rotation angle was set to a random value, and the defect classification accuracy dropped to 96.2% (only one corner was misclassified as a hole). The overall task time remained unchanged.

[0050] (4) Continuous load test: In 20 cycles of tasks, the target material sorting system performed stably, with a time standard deviation of ±3 seconds, and no positioning failure caused by cumulative error occurred.

[0051] After the above systematic testing, the results show that this embodiment achieved a 100% task completion rate and a 98.2% average classification accuracy, with the optimal task completion time reaching 1 minute and 55 seconds. The core indicators significantly surpass traditional solutions, and the lightweight deployment and hardware co-optimization meet offline operation requirements. All test data was recorded via video and archived via logs to ensure traceability and reproducibility, laying a solid foundation for subsequent technological iterations.

[0052] As can be seen, in this application, once the material to be sorted is placed on the target shelf, the image acquisition and preprocessing actions of the target visual perception device are automatically triggered. After the target material sorting system obtains the target image data corresponding to the material to be sorted, it can call the target classification model and use the target classification model to process the target image data to determine the defect type of the material to be sorted. Finally, the defect type detection result corresponding to the material to be sorted is written to the second PLC controller corresponding to the material sorting equipment, thereby using the material sorting equipment to sort the material to be sorted to the corresponding material sorting area, completing the entire intelligent material sorting process and improving the efficiency and accuracy of material sorting.

[0053] As can be seen from the previous embodiment, this application discloses an intelligent material sorting method, which is applied to a terminal device equipped with a target material sorting system, and can improve the efficiency and accuracy of material sorting. The specific implementation process of the target material sorting system will be described below.

[0054] The implementation of the target material sorting system includes four core parts: image acquisition, algorithm deployment, PLC communication, and front-end interface monitoring.

[0055] In this embodiment, image acquisition includes: connecting the camera to a computer via a network cable and matching the camera's IP address in the MVS software. Then, camera acquisition is performed, camera parameters such as image width, height, and pixel format are set, and the generated dataset is saved for subsequent training. After the algorithm is deployed to the EPC device, the PLC controller will control the three-axis grasping system to grasp the card and place it in the designated location based on the camera's recognition results. Specifically, libraries provided by industrial camera manufacturers can be utilized to calculate the optimal transmission packet size and reduce the risk of frame loss from GigE cameras, such as setting a 2000ms acquisition timeout threshold to automatically retry when the network fluctuates, ensuring continuous acquisition. Figure 6 The image shown is a specific real-time feedback image from a camera. During algorithm design, through continuous iteration, targeted solutions were provided for certain problems: 1. Device hot-plug support: The device list is refreshed periodically by calling MV_CC_EnumDevices; 2. Memory leak prevention: C structure memory is managed using ctypes' byref; 3. Color space conversion: The RGB data output by the camera is converted to the OpenCV standard BGR format.

[0056] In this embodiment, the algorithm deployment can include: developing and managing containerized applications on a Windows system using Docker Desktop combined with WSL2 and PuTTY. WSL2 provides a complete Linux environment, making Linux application development on Windows seamless and efficient. PuTTY is a powerful SSH client that can easily connect to WSL2, package the application and its dependencies into a portable container, and deploy it to the EPC device. The basic process is as follows: pull the Python image into Docker and then create a container. Integrate the written code into the container, run the Docker container, test its local operation, and then push the image to Docker Hub. Connect the computer to the EPC device via Ethernet cable, pull the image, and then run the container. A schematic diagram of the container architecture is shown below. Figure 7 As shown.

[0057] In this embodiment, the implementation of PLC communication may include: using the open-source PyPlcnextRsc library to implement communication between the camera and the PLC. The PyPlcnextRsc library implements the ARP (Automation Runtime Protocol) stack and supports data serialization via the RscVariant type. The system-defined udt_ImageClassify structure maps the PLC's DB block data as follows: TypeStore=DataTypeStore.fromString( " " TYPE udt_ImageClassify:STRUCT index:INT; END_STRUCT END_TYPE " " ); In addition, this embodiment also includes a data synchronization mechanism: It listens for the rising edge of Rsc_xExecute via an event-driven mechanism, and resets the index field after a 2-second delay upon detection, preventing the PLC controller from erroneously triggering multiple actions. A code example is shown below: with Device('192.168.3.99',secureInfoSupplier=lambda:("admin","95bfea89")) as device: data_access = IDataAccessService(device) while True: trigger = data_access.Readsingle("Rsc_xExecute").Value if trigger: data_access.WriteSingle(WriteItem("Rsc_xExecute",False)) #Write the test results Out_write.index = defect_type data_access.Write((writeItem(Out_port_name,RscVariant.of(out_write)),)) time.sleep(2) # Keep the signal for 2 seconds Out_write.index=0#Reset signal; In addition, this embodiment provides targeted solutions for possible anomalies to improve system robustness, including: 1. Heartbeat detection: periodically reading the Rsc_iHeartBeatSnd register to verify the connection status; 2. Disconnection and reconnection: automatically handling TCP connection anomalies within the Device context manager; 3. Data type verification: strictly matching PLC variable types using the RscType enumeration.

[0058] In this embodiment, the front-end interface monitoring function operates as follows: 1. Device initialization: Move the three axes to the initial safe position and check whether the status of each device meets the operating conditions; 2. Automatic operation: After initialization, click the automatic operation button on the eHMI screen to enter the automatic operation mode. Then press the start button, and the device will start running. At this time, place the object to be tested on the shelf, and then automatically take pictures and recognize it, and use the three-axis gripper to grab the item to the defined shelf position. Note that the items to be recognized by the visual algorithm need to be defined in advance, corresponding to shelf positions 1-12. After placement, the Y-axis returns to the safe position. At this time, if you continue to place the object to be tested, it will be automatically recognized and continue to run. The example diagram of the front-end monitoring webpage is shown below. Figure 8 As shown.

[0059] As can be seen, this embodiment focuses on the core modules of the target material sorting system, including four main parts: image acquisition, algorithm deployment, PLC communication, and front-end monitoring interface. It details the engineering implementation process and solutions to key technical challenges. Through high-level hardware and software collaboration, the target material sorting system can efficiently and stably complete identification and automatic grasping tasks, supporting industrial-grade automated visual sorting scenarios.

[0060] See Figure 9 As shown, this application discloses an intelligent material sorting device, applied to a terminal device equipped with a target material sorting system, comprising: The image data acquisition module 11 is used to acquire target image data sent by the target visual sensing device corresponding to the target material sorting system; the target image data is the image data obtained by the target visual sensing device after acquiring the target acquisition signal, acquiring the original image data corresponding to the material to be sorted, and preprocessing the original image data; the target acquisition signal is the PLC signal triggered by the first PLC controller corresponding to the target platform after the material to be sorted is placed on the target platform; The defect type detection module 12 is used to call the target classification model through the first target interface of the target material sorting system, so as to use the target classification model to obtain the defect type detection result corresponding to the material to be sorted based on the target image data; The first material sorting module 13 is used to write the defect type detection result into the second PLC controller corresponding to the material sorting equipment of the target material sorting system, so as to use the second PLC controller to control the material sorting equipment to sort the material to be sorted to the corresponding material storage area based on the defect type detection result.

[0061] As can be seen, in this application, once the material to be sorted is placed on the target shelf, the image acquisition and preprocessing actions of the target visual perception device are automatically triggered. After the target material sorting system obtains the target image data corresponding to the material to be sorted, it can call the target classification model and use the target classification model to process the target image data to determine the defect type of the material to be sorted. Finally, the defect type detection result corresponding to the material to be sorted is written to the second PLC controller corresponding to the material sorting equipment, thereby using the material sorting equipment to sort the material to be sorted to the corresponding material sorting area, completing the entire intelligent material sorting process and improving the efficiency and accuracy of material sorting.

[0062] In one specific embodiment, the device may further include: The preliminary fitting module is used to extract the contours of the noise data in the original image data using a target contour fitting algorithm to obtain preliminary fitting results. The fitting optimization module is used to optimize the preliminary fitting result based on the target constraints to obtain the target fitting result corresponding to the original image data; the target constraints include radius constraints determined based on the size of the material to be sorted. The image data generation module is used to filter the original image data based on the target fitting result to obtain filtered image data, and to use a target region segmentation algorithm to determine the candidate defect region corresponding to the filtered image data, so as to obtain the target image data corresponding to the material to be sorted based on the candidate defect region and the filtered image data.

[0063] In one specific embodiment, the device may further include: The condition judgment module is used to determine whether there are noise points in the filtered image data and whether the current window size of the filtering window has reached the maximum window limit. A window adjustment module is used to adjust the filter window based on a dynamic window adjustment strategy to obtain the adjusted filter window if there are noise points in the filtered image data and the filtered window has not reached the maximum window limit. The filtering module is used to filter the filtered image data based on the adjusted filtering window and the target fitting result to obtain new filtered image data, and then jump to the step of determining whether there are noise points in the filtered image data.

[0064] In one specific embodiment, the device may further include: The requirement elicitation module is used to obtain the target material classification requirements; The classification model determination module is used to determine the target classification model from the target model pool based on the target material classification requirements; The target model pool includes a first classification model based on a lightweight network architecture, a second classification model based on a residual network architecture, and a third classification model based on a Transformer architecture.

[0065] In one specific embodiment, the defect type detection module 12 may include: The gradient information determination unit is used to determine the model channel weights of the target classification model based on a local cross-channel interaction strategy if the target classification model is the first classification model based on a lightweight network architecture, so as to determine the target gradient information based on the model channel weights. The classification model update unit is used to obtain the defect type detection result corresponding to the material to be sorted based on the target image data using the target classification model, and when the acquisition time of the defect type detection result is greater than a preset inference time threshold, to adjust the model channel weights of the target classification model based on the target gradient information using a target channel pruning strategy to obtain the updated target classification model.

[0066] In one specific embodiment, the defect type detection module 12 may include: The feature determination unit is used to determine the target geometric features and target grayscale features corresponding to the candidate defect region based on the target image data using the target classification model. The detection result acquisition unit is used to determine the defect type corresponding to the candidate defect region based on the target geometric features and the target grayscale features, and to obtain the defect type detection result corresponding to the material to be sorted based on the defect type corresponding to each candidate defect region of the target image data.

[0067] In one specific embodiment, the device may further include: The second material sorting module is used to transmit the video data collected by the target visual perception device to the front-end interface corresponding to the target material sorting system in real time, and obtain material sorting instructions through the front-end interface, so as to sort the materials to be sorted on the target platform based on the material sorting instructions. The material sorting instructions include sorting start instructions, sorting stop instructions, and sorting mode adjustment instructions.

[0068] Furthermore, embodiments of this application also disclose an electronic device, Figure 10 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0069] Figure 10 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the intelligent material sorting method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0070] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0071] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored thereon can include an operating system 221, computer programs 222, etc., and the storage method can be temporary storage or permanent storage.

[0072] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the intelligent material sorting method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0073] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned intelligent material sorting method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0074] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0075] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0076] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0077] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0078] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An intelligent material sorting method, characterized in that, Terminal equipment used in systems for sorting target materials includes: The system acquires target image data sent by the target visual sensing device corresponding to the target material sorting system; the target image data is the original image data corresponding to the material to be sorted acquired by the target visual sensing device after acquiring the target acquisition signal, and the image data is obtained after preprocessing the original image data; the target acquisition signal is the PLC signal triggered by the first PLC controller corresponding to the target placement platform after the material to be sorted is placed on the target placement platform; The target classification model is invoked using the first target interface of the target material sorting system, so as to obtain the defect type detection result corresponding to the material to be sorted based on the target image data using the target classification model; The defect type detection result is written into the second PLC controller corresponding to the material sorting equipment of the target material sorting system, so that the second PLC controller can control the material sorting equipment to sort the material to be sorted to the corresponding material storage area based on the defect type detection result.

2. The intelligent material sorting method according to claim 1, characterized in that, The process of preprocessing the original image data includes: A target contour fitting algorithm is used to extract contours from the noise data in the original image data to obtain preliminary fitting results. The preliminary fitting result is optimized based on the target constraints to obtain the target fitting result corresponding to the original image data; the target constraints include radius constraints determined based on the size of the material to be sorted. The original image data is filtered based on the target fitting result to obtain filtered image data, and a target region segmentation algorithm is used to determine the candidate defect region corresponding to the filtered image data, so as to obtain the target image data corresponding to the material to be sorted based on the candidate defect region and the filtered image data.

3. The intelligent material sorting method according to claim 2, characterized in that, Before determining the candidate defect region corresponding to the filtered image data using the target region segmentation algorithm, the method further includes: Determine whether there are noise points in the filtered image data, and determine whether the current window size of the filtering window has reached the maximum window limit; If there are noise points in the filtered image data and the filtering window has not reached the maximum window limit, the filtering window is adjusted based on a dynamic window adjustment strategy to obtain the adjusted filtering window. The filtered image data is filtered based on the adjusted filtering window and the target fitting result to obtain new filtered image data, and then the process jumps to the step of determining whether there are noise points in the filtered image data.

4. The intelligent material sorting method according to claim 1, characterized in that, Before calling the target classification model using the first target interface of the target material sorting system, the method further includes: Obtain the target material classification requirements; Based on the target material classification requirements, a target classification model is determined from the target model pool; The target model pool includes a first classification model based on a lightweight network architecture, a second classification model based on a residual network architecture, and a third classification model based on a Transformer architecture.

5. The intelligent material sorting method according to claim 4, characterized in that, The step of using the target classification model to obtain the defect type detection result corresponding to the material to be sorted based on the target image data includes: If the target classification model is the first classification model based on a lightweight network architecture, then the model channel weights of the target classification model are determined based on a local cross-channel interaction strategy, so as to determine the target gradient information based on the model channel weights; The target classification model is used to obtain the defect type detection result corresponding to the material to be sorted based on the target image data. When the acquisition time of the defect type detection result is greater than the preset inference time threshold, the target channel pruning strategy is used to adjust the model channel weight of the target classification model based on the target gradient information to obtain the updated target classification model.

6. The intelligent material sorting method according to claim 1, characterized in that, The step of using the target classification model to obtain the defect type detection result corresponding to the material to be sorted based on the target image data includes: The target classification model is used to determine the target geometric features and target grayscale features corresponding to the candidate defect regions based on the target image data; Based on the target geometric features and the target grayscale features, the defect type corresponding to the candidate defect region is determined, and the defect type detection result corresponding to the material to be sorted is obtained based on the defect type corresponding to each candidate defect region of the target image data.

7. The intelligent material sorting method according to any one of claims 1 to 6, characterized in that, Also includes: The video data collected by the target visual perception device is transmitted in real time to the front-end interface corresponding to the target material sorting system, and the material sorting instructions are obtained through the front-end interface to sort the materials to be sorted on the target platform based on the material sorting instructions. The material sorting instructions include sorting start instructions, sorting stop instructions, and sorting mode adjustment instructions.

8. An intelligent material sorting device, characterized in that, Terminal equipment used in systems for sorting target materials includes: The image data acquisition module is used to acquire target image data sent by the target visual sensing device corresponding to the target material sorting system; the target image data is the original image data corresponding to the material to be sorted acquired by the target visual sensing device after acquiring the target acquisition signal, and the image data is obtained after preprocessing the original image data; the target acquisition signal is the PLC signal triggered by the first PLC controller corresponding to the target platform after the material to be sorted is placed on the target platform; The defect type detection module is used to call the target classification model through the first target interface of the target material sorting system, so as to use the target classification model to obtain the defect type detection result corresponding to the material to be sorted based on the target image data; The first material sorting module is used to write the defect type detection result into the second PLC controller corresponding to the material sorting equipment of the target material sorting system, so as to use the second PLC controller to control the material sorting equipment to sort the material to be sorted to the corresponding material storage area based on the defect type detection result.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the intelligent material sorting method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the intelligent material sorting method as described in any one of claims 1 to 7.

Images