Anode appearance online detection and sorting method and system based on multi-sensor fusion and artificial intelligence and storage medium

Abstract

The invention relates to the technical field of industrial automation, and particularly discloses an anode appearance online detecting and sorting method and system based on multi-sensor fusion and artificial intelligence and a storage medium. The method comprises the steps that when a photoelectric sensor detects that a material is in place, a soft trigger command is sent to a designated camera through an upper computer industrial personal computer, and the camera is started; the camera collects image data; after acquiring camera image data, the upper computer industrial personal computer sends the camera image data to the Python server through a TCP (Transmission Control Protocol); the Python server calls a YOLO model to perform instance segmentation and feature extraction, executes spatial position judgment, generates an information text and a visual result map, and transmits the information text and the visual result map back to the upper computer industrial personal computer; after the material leaves the laser measurement area, the upper computer industrial personal computer outputs the final height of the material; the upper computer industrial personal computer obtains the category and the position state of each instance; and in combination with the category and the position state of each instance and the final height of the corresponding material, comparing with a preset quality standard, and judging whether the corresponding material needs to be sorted or not.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of industrial automation, in particular to an anode appearance online detection and sorting method and system based on multi-sensor fusion and artificial intelligence and a storage medium. BACKGROUND

[0002] In industrial production, online quality detection and automatic sorting of products are key links to ensure product quality and improve production efficiency. Taking carbon block production in the carbon industry as an example, 100% inspection of the size, appearance defects (such as cracks, corner drop) of carbon blocks is required, and unqualified products are automatically removed.

[0003] At present, the common detection scheme mainly has the following shortcomings: 1. Manual detection: relying on workers' naked eye observation and caliper measurement, which has problems such as low efficiency, high labor intensity, strong subjectivity, easy to miss detection and difficult to form digital records.

[0004] 2. Traditional machine vision: using conventional image processing algorithms (such as edge detection, threshold segmentation) for two-dimensional size measurement and defect recognition. This method is sensitive to light changes, has poor segmentation effect on complex backgrounds and objects adhering to each other, and is difficult to distinguish different types of defects, with weak generalization ability.

[0005] 3. Single sensor detection: using only cameras or only laser sensors. Two-dimensional vision lacks depth information and cannot accurately measure the height or flatness of objects; and single laser ranging can only provide single-point or single-line height information, and cannot obtain the overall topography and surface quality of the object.

[0006] In recent years, deep learning technology, especially the target detection and instance segmentation model (such as YOLO, MaskR-CNN) based on convolutional neural network (CNN), has shown great advantages in image recognition. They can accurately locate, classify and segment multiple object instances from complex backgrounds. However, the deep fusion of deep learning models and traditional industrial automation devices (such as PLC, laser range finder) to build a stable, real-time, end-to-end intelligent detection and sorting system still faces many challenges: High system integration complexity: needs to solve the problems of driving, synchronization and data fusion of different hardware (cameras, lasers, PLCs).

[0007] Real-time requirement: the production line has a fast pace, and the entire closed-loop response time from image acquisition, processing to execution of sorting must be in milliseconds.

[0008] Robustness of data processing: the industrial site environment is harsh, and sensor data has noise and jitter, which requires the design of special algorithms to ensure the stability and accuracy of the measurement results.

[0009] Comprehensive judgment logic: a single judgment standard is often insufficient to make accurate sorting decisions, and multiple information (such as appearance defects, size, positional relationship) needs to be integrated for collaborative judgment.

[0010] Therefore, there is an urgent need in the art for a high-precision, high-robustness online detection and sorting solution that can integrate multi-sensor information, utilize advanced artificial intelligence algorithms, and seamlessly integrate with industrial control systems. SUMMARY

[0011] The purpose of the present application is to provide an anode appearance online detection and sorting method and system based on multi-sensor fusion and artificial intelligence, to solve the problems in the prior art.

[0012] To achieve the above-mentioned purpose, the embodiments of the present application provide an anode appearance online detection and sorting method based on multi-sensor fusion and artificial intelligence, comprising: Run the WPF host computer application, initialize the camera SDK, start the TCP server, and connect the PLC controller and the laser range finder; The PLC controller continuously monitors the photoelectric sensor of the production line. When the photoelectric sensor detects that the material is in place, it sends a soft trigger command to the designated camera through the host computer, and the camera acquires image data; After the host computer acquires camera image data, it sends it to the Python server via TCP; The Python server calls the YOLO model for instance segmentation and feature extraction, performs spatial position judgment, generates information text and visualization result images, and returns them to the host computer; When the material leaves the laser measurement area, the host computer executes a multi-stage processing algorithm on the material height difference data array sent by the laser range finder, and outputs the final height of the material; The host computer parses the information text returned by the Python server to obtain the category and position state of each instance; Combine the category and position state of each instance with the corresponding final height of the material, and compare it with the preset quality standard to determine whether the corresponding material needs to be sorted.

[0013] Optionally, after determining whether the corresponding material needs to be sorted, it further comprises: If it is determined that the corresponding material is a defective product that needs to be sorted, the host computer writes a `True` signal to the specific output point of the PLC controller through the S7.Net library; After receiving the signal, the PLC controller drives the actuator to push the defective product out of the production line.

[0014] Optionally, the host computer industrial computer executes a multi-stage processing algorithm on the material height difference value data array sent by the laser range finder, and outputs the final height of the material, specifically including: Dynamic caching and rising phase detection: during the detection of the material by the laser range finder, continuously store the height difference value into the array `TKGD`, find the first stable starting point in the data sequence of the array `TKGD` that satisfies the condition that the fluctuation of the three consecutive points is less than a preset threshold, and eliminate all rising phase data before the point; Median-based outlier filtering: calculate the median of the stable phase data after the stable starting point, and based on the median, set a dynamic range, and consider all data outside the dynamic range after the stable starting point as outliers and eliminate them; Robust final value calculation: calculate the median of the valid data sequence obtained after filtering again as the final height of the material.

[0015] Optionally, the category and position state of each instance and the corresponding final height of the material are combined, and compared with a preset quality standard to determine whether the corresponding material needs to be sorted, specifically including: Identify all instance bounding boxes of the category "TKA", and then check whether the instance bounding box of the waste block is completely contained in a certain "TKA" bounding box to obtain a first determination result; Determine whether the final height of the corresponding material exceeds a preset tolerance range to obtain a second determination result; Determine whether the corresponding material needs to be sorted by taking the logical "AND" or "OR" of the first determination result and the second determination result.

[0016] To achieve the above purpose, the application also provides an anode appearance online detection and sorting system based on multi-sensor fusion and artificial intelligence, comprising: a host computer industrial computer, a camera array, a laser range finder, a PLC controller and a photoelectric sensor, wherein, The host computer industrial computer is connected to the camera array, the laser range finder and the PLC controller, and the PLC controller is connected to the photoelectric sensor; The camera array, the laser range finder and the photoelectric sensor are arranged on a steel frame arranged above a conveying line, and the conveying line is used to convey carbon blocks; The host computer industrial computer has a built-in WPF application, including a camera control module, an image processing and transmission module, a network communication module, a laser ranging communication module, a PLC communication module and a core logic judgment module, wherein, The camera control module encapsulates the camera SDK, which is used to realize the enumeration, parameter configuration, soft trigger acquisition, image cache acquisition and release of the camera; The image processing and transmission module is used to convert the original image data collected by the camera into a format, and package and send the converted image data and its camera index to the Python server; The network communication module includes TCP server and client functions; The laser ranging communication module establishes a connection with the laser range finder through TcpClient, sends measurement instructions, and asynchronously receives the returned raw data; The PLC communication module uses the S7.Net library to communicate with the PLC controller, and through the `Plc` class object, it realizes the read-write operation of the internal relays of the PLC controller; The core logic judgment module is built-in image result analysis and position judgment algorithm and laser ranging data multi-stage processing algorithm, which is used to compare the final height of each instance with the preset quality standard, to determine whether the corresponding material needs to be sorted.

[0017] Optionally, the image result analysis and position judgment algorithm specifically includes: From the string returned by the Python server, use regular expressions to parse the detailed information of each instance, identify all instance bounding boxes of the "TKA" category, and then check whether the instance bounding box of the waste block is completely contained in a certain "TKA" bounding box to obtain the first judgment result.

[0018] Optionally, the laser ranging data multi-stage processing algorithm specifically includes: Dynamic caching and rising stage detection: During the detection of the material by the laser range finder, continuously store the height difference value in the array `TKGD`, find the first stable starting point in the data sequence of the array `TKGD` that satisfies the condition that the fluctuation of the three consecutive points is less than the preset threshold, and eliminate all rising stage data before the point; Median-based outlier filtering: Calculate the median of the stable stage data after the stable starting point, and based on the median, set a dynamic range, and consider all data outside the dynamic range after the stable starting point as outliers and eliminate them; Robust final value calculation: Calculate the median of the valid data sequence obtained after filtering again as the final height of the material; Determine whether the final height of the corresponding material exceeds the preset tolerance range to obtain the second judgment result.

[0019] Optionally, it also includes: A data management and visualization module is used to display the input / output images of the camera, dynamic graphs of laser measurement data, data tables, and system logs in real time, and a LiveCharts control is used to draw a curve of the height of the carbon block over time, with a target line and a tolerance line superimposed.

[0020] Optionally, the steel frame is erected above the conveying line and includes a stand, a first cross beam, a second cross beam, a third cross beam, a fourth cross beam, a first longitudinal beam, a second longitudinal beam, a third longitudinal beam, a fourth longitudinal beam, and a fifth longitudinal beam. The two ends of the first cross beam, the second cross beam, the first longitudinal beam, and the second longitudinal beam are connected to each other and to the top end of the stand. The third cross beam is arranged in parallel below the first cross beam, and the fourth cross beam is arranged in parallel below the second cross beam. The third longitudinal beam is arranged in parallel below the first longitudinal beam, the fourth longitudinal beam is arranged in parallel between the first longitudinal beam and the second longitudinal beam, and the fifth longitudinal beam is arranged in parallel below the third longitudinal beam. The camera is provided on the third longitudinal beam, the fourth longitudinal beam, the fifth longitudinal beam, the third cross beam, and the fourth cross beam. The laser range finder is arranged on the third longitudinal beam, and the photoelectric sensor is arranged on the third cross beam and the fourth cross beam.

[0021] A processor connected to the memory, the processor being configured to perform the steps of the method as described above.

[0022] To achieve the above-mentioned purpose, the application also provides a computer storage medium, which stores a computer program, wherein the computer program is executed by a machine to realize the steps of the method as described above.

[0023] The embodiments of the application have the following advantages: 1. Comprehensive detection of dimensions: Combining 2D visual appearance detection and 1D laser precise ranging, it can not only identify surface defects and types, but also obtain key height dimensions, realizing multi-dimensional comprehensive evaluation of material quality.

[0024] 2. High detection accuracy and robustness: Sub-pixel edge detection algorithm and temperature compensation laser ranging technology are adopted to effectively overcome the problems of environmental light interference and thermal drift, ensuring long-term stable operation of high-precision measurement in complex industrial environments.

[0025] The instance segmentation model (YOLOv8-seg) based on deep learning is adopted, which has strong adaptability to complex backgrounds, light changes, and object deformation compared with traditional algorithms, and the segmentation accuracy and recognition accuracy are greatly improved.

[0026] The innovative multi-stage processing algorithm for laser ranging data effectively filters out unstable data and random outliers in the early stages of measurement. By calculating the median, it ensures the reliability and repeatability of the final height value and avoids misjudgments caused by single-point jitter or instantaneous interference.

[0027] The calculation of instance length and width using the minimum bounding rectangle is more accurate than the traditional horizontal bounding box, reflecting the true geometric features of the object.

[0028] 3. High degree of automation and intelligence: It has achieved fully automated operation of the entire process from material triggering, image acquisition, AI analysis, data fusion, logical judgment to automatic sorting, which greatly liberates manpower and improves production efficiency.

[0029] The introduction of spatial logic judgment based on TKA container enables the system to understand the relationships between objects in the scene, achieving a higher level of semantic detection, rather than just simple existence detection.

[0030] 4. High system integration and good real-time performance: Through a carefully designed software architecture and communication protocol, heterogeneous hardware devices (various brands of cameras, laser sensors, PLCs) and software modules (C# WPF, Python AI services) are seamlessly integrated into a stable and unified system.

[0031] By employing technologies such as multithreading, asynchronous programming, and thread-safe queues (`ConcurrentQueue`), parallel execution of tasks such as image acquisition, network communication, and data display is ensured, meeting the stringent real-time requirements of industrial sites.

[0032] 5. High maintainability and scalability: The modular design allows each part to operate relatively independently. For example, when upgrading an AI model, only the model file on the Python server needs to be replaced, without modifying the core code on the host computer.

[0033] The clear TCP / IP communication interface facilitates system expansion, allowing for easy addition of more cameras, sensors, or data interfacing with other MES (Manufacturing Execution Systems).

[0034] Comprehensive logging and data storage capabilities provide a solid data foundation for fault diagnosis, process optimization, and quality traceability. Attached Figure Description

[0035] To more clearly illustrate the embodiments of this application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0036] Figure 1 A schematic diagram of the overall hardware architecture and connection of an online inspection and sorting system for anodized appearance based on multi-sensor fusion and artificial intelligence, provided for at least one embodiment of this application; Figure 2 A flowchart illustrating an online inspection and sorting method for anodized appearance based on multi-sensor fusion and artificial intelligence, provided for at least one embodiment of this application; Figure 3 A schematic diagram of the camera image display area of ​​the main interface of the WPF host computer software for an online inspection and sorting method of anode appearance based on multi-sensor fusion and artificial intelligence, provided for at least one embodiment of this application; Figure 4 A schematic diagram of laser data charts on the main interface of WPF host computer software for an online detection and sorting method for anode appearance based on multi-sensor fusion and artificial intelligence, provided for at least one embodiment of this application; Figure 5 A schematic diagram of the control buttons and log window of the main interface of the WPF host computer software for an online detection and sorting method for anod appearance based on multi-sensor fusion and artificial intelligence, provided for at least one embodiment of this application; Figure 6 A software flowchart illustrating the image processing and sorting decision-making of an online anode appearance detection and sorting method based on multi-sensor fusion and artificial intelligence, provided for at least one embodiment of this application; Figure 7 A detailed flowchart of a multi-stage processing algorithm for laser ranging data in an online detection and sorting method for anode appearance based on multi-sensor fusion and artificial intelligence, provided for at least one embodiment of this application; Figure 8 An example image showing the visualization results of a YOLO model processed for an online inspection and sorting method of anodized appearance based on multi-sensor fusion and artificial intelligence, provided for at least one embodiment of this application; Figure 9 A timing diagram of the input / output signal interaction between a PLC controller and an online anod appearance inspection and sorting system based on multi-sensor fusion and artificial intelligence, provided for at least one embodiment of this application; Figure 10This is a hardware installation diagram of an online anodizing appearance inspection and sorting system based on multi-sensor fusion and artificial intelligence, provided for at least one embodiment of this application. Attached image description: 1. Column, 2. First crossbeam, 3. Second crossbeam, 4. Third crossbeam, 5. Fourth crossbeam, 6. First longitudinal beam, 7. Second longitudinal beam, 8. Third longitudinal beam, 9. Fourth longitudinal beam, 10. Fifth longitudinal beam, 11. Conveyor line, 12. Carbon block, 13. Camera, 14. Lighting lamp, 15. Laser rangefinder, 16. Photoelectric sensor. Detailed Implementation

[0038] The following specific embodiments illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0039] In the description of this application, the terms “center,” “upper,” “lower,” “left,” “right,” “vertical,” “horizontal,” “inner,” and “outer,” etc., indicating orientation or positional relationships, are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0040] Unless otherwise expressly specified and limited, the terms "set up," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0041] It should be noted that the steps in the claims and description of this application may be performed substantially in parallel or in reverse order where appropriate, depending on the function involved. The terms "first," "second," "third," etc. are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0042] It should also be noted that the terms "step S1", "step S2", "step S3", etc. in the claims and description of this application are used to distinguish different steps, rather than to describe a specific order or sequence. It should be understood that these steps may be performed substantially in parallel or in reverse order under appropriate circumstances, depending on the function involved.

[0043] Furthermore, the technical features involved in the different embodiments of this application described below can be combined with each other as long as they do not conflict with each other.

[0044] One embodiment of this application provides an online inspection and sorting system for anodized appearance based on multi-sensor fusion and artificial intelligence. This system employs a layered design, referencing... Figure 1 Mainly includes: Sensing Layer: Composed of an array of eight high-resolution cameras, responsible for acquiring high-definition images of the material from different angles; a high-precision laser rangefinder is responsible for measuring the height of the material relative to a reference surface in real time. The industrial cameras can use an RJ45 interface and support software triggering to ensure synchronization. The laser rangefinder communicates with the host computer via TCP / IP protocol to transmit measurement data in real time.

[0045] The transmission and processing layer includes a host computer / industrial control computer and a dedicated image processing server. The host computer, developed using the C# WPF framework, serves as the system's control hub, responsible for controlling the camera and laser rangefinder, communicating with the PLC, human-machine interaction, and handling core business logic. The image processing server runs a Python environment, loading a pre-trained YOLOv8 instance segmentation model, and is responsible for high-performance inference computation on the received images. Both layers exchange data at high speed via a custom TCP / IP protocol.

[0046] Execution and Control Layer: An industrial-grade PLC controller (such as a Siemens S7-200Smart) is used as the lower-level machine. It receives sorting instructions from the upper-level machine and drives pneumatic or electric actuators to complete the sorting action. Simultaneously, the PLC controller is also connected to photoelectric sensors to detect the arrival status of materials and provide trigger signals for image acquisition and laser measurement.

[0047] The host computer / industrial control computer is connected to the camera array, the laser rangefinder, and the PLC controller, respectively. The PLC controller is connected to the photoelectric sensor. (Reference) Figure 9 .

[0048] The camera array 13, laser rangefinder 15, and photoelectric sensor 16 are mounted on a steel frame above the conveyor line 11, which is used to transport carbon blocks 12. (Reference) Figure 10 .

[0049] In some embodiments, the steel frame is erected above the conveyor line 11 and includes a column 1, a first crossbeam 2, a second crossbeam 3, a third crossbeam 4, a fourth crossbeam 5, a first longitudinal beam 6, a second longitudinal beam 7, a third longitudinal beam 8, a fourth longitudinal beam 9, and a fifth longitudinal beam 10, wherein: The two ends of the first crossbeam 2, the second crossbeam 3, the first longitudinal beam 6, and the second longitudinal beam 7 are connected to each other and respectively connected to the top of the column 1; The third crossbeam 4 is arranged parallel to the bottom of the first crossbeam 2, and the fourth crossbeam 5 is arranged parallel to the bottom of the second crossbeam 3. The third longitudinal beam 8 is arranged parallel to the bottom of the first longitudinal beam 6, the fourth longitudinal beam 9 is arranged parallel to the middle of the first longitudinal beam 6 and the second longitudinal beam 7, and the fifth longitudinal beam 10 is arranged parallel to the bottom of the third longitudinal beam 8.

[0050] In some embodiments, there are 8 cameras 13, with two cameras 13 respectively installed on the third longitudinal beam 8, the fourth longitudinal beam 9, and the fifth longitudinal beam 10, and one camera 13 respectively installed on the third crossbeam 4 and the fourth crossbeam 5.

[0051] In some embodiments, the laser rangefinder 15 is disposed on the fourth longitudinal beam 9, and the photoelectric sensor 16 is disposed on the third crossbeam 4 and the fourth crossbeam 5.

[0052] In some embodiments, the steel frame is also provided with a lighting lamp 14.

[0053] In some embodiments, lighting lamps 14 are provided on the first longitudinal beam 6, the second longitudinal beam 7 and the third longitudinal beam 8 respectively.

[0054] The host industrial control computer has a built-in WPF application, including a camera control module, an image processing and transmission module, a network communication module, a laser ranging communication module, a PLC communication module, and a core logic judgment module, specifically: (1) WPF host computer / industrial control computer application Camera control module: This module encapsulates the Hikvision camera SDK. It implements camera enumeration, parameter configuration, soft-trigger acquisition, and image buffer acquisition and release. It supports parallel operation of multiple cameras by managing multiple camera instances. (Refer to...) Figure 3 .

[0055] The image processing and transmission module is responsible for converting the raw image data (Mono8 and BGR8 formats) acquired by the camera into two formats: a) for real-time display in the WPF interface; and b) JPEG byte streams for network transmission and local storage. Thread-safe image queue management is implemented using `ConcurrentQueue`, with a separate background thread (`SendWorker`) responsible for packaging and sending the converted image data and its camera index to the Python server.

[0056] The network communication module includes TCP server and client functionalities. As a server, it listens on a specific port 5951, waiting for Python server client connections, and handles the sending of image data and the receiving of processed results. The communication protocol is rigorously designed, with a data packet structure of: `[camera index (4 bytes)][information length (4 bytes)][information text][image data length (4 bytes)][image data]`, ensuring the integrity and parsability of data transmission.

[0057] The laser ranging communication module is implemented in the `Ruler` and `Ruler_2` classes. It establishes a connection with the laser rangefinder via TcpClient, sends measurement commands ("iACM"), and asynchronously receives the returned raw data. This module includes a data parsing function (`ConvertToMillimeters`) that parses the string returned by the instrument into integer values ​​in millimeters. (See reference.) Figure 4 and Figure 5 .

[0058] PLC Communication Module: Uses the S7.Net library to communicate with Siemens PLC controllers. Through `Plc` class objects, it enables read and write operations on internal relays (M points) of the PLC controller. For example, it reads the status of photoelectric sensors (M0.1, M0.3, M2.0), controls lighting (M0.2), and triggers push-block solenoid valves (M0.6, M3.0).

[0059] Core logic judgment module: This is the core of this application's intelligence. It resides in the `MainWindow` and `Ruler` classes. (See reference) Figure 6 .

[0060] Image result parsing and location determination algorithm: From the string returned by the Python server, regular expressions are used to parse out detailed information for each instance (category, confidence level, area, dimensions, bounding box, etc.). Innovatively, it identifies the bounding boxes of all instances categorized as "TKA" and then checks whether the bounding box of a dead block ("LW") is completely contained within a "TKA" bounding box, obtaining the first determination result. If so, it is marked as "YES" (within the TKA region); otherwise, it is marked as "NO". This logic is crucial for determining whether a defect is located within the valid region of the container.

[0061] The Python server runs the YOLO instance segmentation model, which can segment carbon blocks into different defect categories. Finally, it outputs the category, confidence score, pixel area, pixel length, pixel width, bounding box coordinates, and mask information. If the bounding box coordinates of the category are within the coordinate range of TKA, it is judged as a waste block; otherwise, it is a good block.

[0062] Multi-stage processing algorithm for laser ranging data: To address the jitter and instability issues in laser data when materials first enter the measurement area, this application designs an efficient data cleaning algorithm. (Reference) Figure 7 .

[0063] Phase 1: Dynamic caching and rising phase detection. While the material is being detected by the laser rangefinder, the height difference is continuously stored in the array `TKGD`. Instead of simply averaging all the data, the algorithm uses the `FindStableStartIndex` method to find the first stable starting point in the data sequence of the `TKGD` array where three consecutive fluctuations are less than a threshold (5mm), and then removes all rising phase data before that point.

[0064] Phase Two: Outlier Filtering Based on Median. Calculate the median of the data in the stable phase after the stable starting point, and set a reasonable dynamic range (median ± 15 mm) based on this median. All data exceeding this dynamic range after the stable starting point are considered outliers and removed. This method is more resistant to the interference of extreme outliers than mean-based filtering.

[0065] Phase Three: Robust Final Value Calculation. For the filtered valid data sequence, the median is recalculated as the final height of the material. The median statistic is insensitive to a few outliers and better represents the central tendency of the data, thus obtaining a more reliable height measurement. Finally, it is determined whether the final height of the material exceeds the preset tolerance range, yielding a second judgment result.

[0066] Comprehensive Decision: The final sorting decision is a logical AND or OR operation of the first and second judgment results mentioned above (depending on specific process requirements). For example, the rule could be: "If the material is identified as 'LW' (waste block) and its status is 'YES' (within the valid area), or its height measurement is outside the tolerance range, then sorting is performed." Data Management and Visualization Module: Provides a rich UI interface, displaying real-time input / output images from eight cameras, dynamic curves of laser measurement data, data tables, system logs, etc. It uses the LiveCharts control to plot the curve of carbon block height changing over time, overlaying target lines and tolerance lines to intuitively reflect the production quality status. All operations and inspection results are recorded to a timestamped log file using the `outdatatxt` method for easy traceability and analysis.

[0067] The Python server has a built-in AI analysis and processing program, specifically: (2) Python image processing service YOLO Model Inference Engine: Utilizes the UltralyticsYOLOv8 framework, loading a custom-trained instance segmentation model (`best.pt`). This model outputs bounding boxes, class confidence scores, and pixel-level masks for each detected object.

[0068] Advanced feature extraction: In the `process_and_get_info` function, not only is basic information obtained, but OpenCV is also used to post-process the output mask of the model.

[0069] Find the maximum contour of the mask using `cv2.findContours`.

[0070] Use `cv2.minAreaRect` to calculate the minimum bounding rectangle of the contour. This rectangle more accurately reflects the actual length and width of the object, especially when the object is tilted in the image. The longer side of the rectangle is defined as the object's "length," and the shorter side is defined as its "width," which is more accurate than using the width and height of a horizontal bounding box.

[0071] Spatial relationship determination: On the Python side, all detection results are pre-traversed to collect bounding boxes for all "TKA" categories. Then, when processing instances of other categories, it is checked whether their bounding boxes fall completely within any "TKA" bounding box, and this first determination result ("YES" / "NO") is directly appended to the returned information text, reducing the computational burden on the host computer.

[0072] Communication Protocol Client: As a TCP client, it actively connects to the WPF host computer. It strictly follows the agreed protocol to parse data packets from the host computer (first read the index, then the image length, and finally the image data), and packages the processed results (information text + JPEG image) back in the specified format.

[0073] refer to Figure 2 , Figure 2 The flowchart illustrates an online anodizing appearance inspection and sorting method based on multi-sensor fusion and artificial intelligence, provided in at least one embodiment of this application. It should be understood that this method may further include additional boxes not shown and / or the boxes shown may be omitted; the scope of this application is not limited in this respect. The working method of this application is as follows: 1. System Startup and Initialization: Run the WPF application, initialize the camera SDK, start the TCP server (port 5951), and connect the PLC controller and laser rangefinder. Simultaneously, the Python server starts, loads the YOLO model, and connects to the WPF TCP server.

[0074] 2. Waiting for trigger signal: The system is in standby mode. The PLC controller continuously monitors the photoelectric sensors on the production line.

[0075] 3. Material arrival and data collection: When the photoelectric sensor (M1.3) detects that the material has arrived, the PLC controller informs the host computer of this status.

[0076] The host computer sends a soft trigger command (`MV_CC_SetCommandValue_NET("TriggerSoftware")`) to the designated camera, and the camera acquires image data. Simultaneously, the laser rangefinder continues measurement in a separate thread, and the host computer begins caching material height data.

[0077] 4. Image Processing and AI Analysis: After the host computer / industrial control computer acquires the camera image data, it adds it to the transmission queue.

[0078] The sending thread packages the image data and camera index and sends them to the Python server via TCP.

[0079] The Python server calls the YOLO model to perform instance segmentation and feature extraction, performs spatial location determination, generates information text and visualization results, and sends them back to the host computer / industrial control computer. (Reference) Figure 8 .

[0080] 5. Laser data processing and height calculation: In the host industrial computer, the laser data processing module runs continuously. When the material leaves the laser measurement area (the photoelectric sensor signal disappears), the data processing algorithm is activated, and a multi-stage laser ranging data processing algorithm is executed on the cached height data array, namely "rising stage rejection" → "outlier filtering" → "median calculation" process, and the final height of the material is output.

[0081] 6. Comprehensive Judgment and Sorting Execution: The host computer / industrial control computer parses the text information returned by the Python server to obtain the category and location status of each instance.

[0082] By combining the category and location status of each instance with the corresponding final height of the material, and comparing it with preset quality standards (such as whether it is a waste block or whether the height is within tolerance), it is determined whether the corresponding material needs to be sorted.

[0083] If the corresponding material is determined to be a defective product that needs to be sorted, the host computer / industrial control computer writes a `True` signal to a specific output point (M0.6) of the PLC controller through the S7.Net library.

[0084] After receiving the signal, the PLC controller drives the actuators such as the pusher cylinder to push the defective products out of the production line.

[0085] 7. Data Recording and Feedback: All images, measurement data, judgment results, and operation instructions throughout the process are saved to the hard drive and updated in real time on the UI interface, providing operators with a monitoring interface and data support for quality analysis.

[0086] This application synchronously triggers the acquisition of material images via a camera array, sends them to the YOLO model via TCP protocol for high-precision instance segmentation and feature extraction, and simultaneously uses a laser rangefinder to acquire material dimensions in real time. It innovatively employs a multi-stage data processing workflow, including spatial positional relationship judgment based on the TKA container, elimination during the rising phase of carbon block height data, and a median filtering algorithm, to achieve accurate determination of material quality and size. Finally, the PLC controls the actuator to complete automatic sorting. This application solves the problems of low efficiency, poor consistency, and insufficient detection dimensions of traditional manual inspection, realizing fully automatic, high-precision, multi-dimensional online inspection and sorting of anode appearance.

[0087] The present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment takes online detection and sorting on a carbon block production line as the application scenario.

[0088] 1. Hardware selection and connection: Cameras: Eight Hikvision MV-CH050-10UM 5-megapixel USB 3.0 industrial cameras were selected and installed above key workstations on the production line, and connected to the host industrial computer via USB hubs.

[0089] Laser rangefinder: The Shendawei series laser rangefinder is selected and connected to the industrial control computer via Ethernet interface, corresponding to the east and west conveyor lines respectively.

[0090] Host computer / industrial PC: An Advantech industrial PC is selected, equipped with an Intel processor, 16GB of memory, a 1TB SSD, and a high-performance dedicated graphics card (for optional GPU acceleration), and Windows 10 operating system and .NET Framework 4.6.1 are installed.

[0091] The PLC controller is a Siemens S7-200SMARTSR20, which communicates with the industrial computer via Ethernet. Its digital input points (I) are connected to Omron E3Z-T81 photoelectric sensors, and its digital output points (Q) control the pusher block's movement.

[0092] Network switch: Uses gigabit industrial Ethernet switches to connect industrial control computers, laser rangefinders and PLCs.

[0093] 2. Software environment deployment: Host computer environment: Install the Visual Studio runtime library and Hikvision MVS camera SDK on the industrial control computer. Deploy the WPF application `MyHKDome.exe` compiled using the C# code described in this application.

[0094] Python server environment: Install Python 3.8, OpenCV, PyTorch, UltralyticsYOLO, and other libraries on the same industrial control computer. Deploy the Python script `yolo_server.py` described in this application and the trained YOLO model file `best.pt`.

[0095] Software Configuration and Startup 1. System startup: After the program starts, it automatically initializes the camera SDK and starts a TCP server listening on port 5951.

[0096] The operator clicks the "Scan Cameras" button on the interface, and the program enumerates 8 connected cameras.

[0097] Click the "Open Camera" button, and the program will create the device, open the device, set the trigger mode to "On" and the trigger source to "Software" in sequence, and start capturing the image stream.

[0098] Click the "Connect PLC" button to establish a connection between the program and the Siemens PLC (IP: 192.168.1.2).

[0099] On the "Laser Rangefinder" interface, click the "Connect" button to establish a TCP connection between the program and the laser rangefinder (IP: 192.168.1.10).

[0100] 2. Parameter settings: In the laser ranging interface, set the target height of the carbon block to 600mm, the reference height to 1500mm, the positive tolerance to 10mm, and the negative tolerance to 10mm.

[0101] Turn on the "Automatic Push Blocks" checkbox to enable the automatic sorting function.

[0102] 3. Online inspection and sorting process: A carbon block enters the testing station via the conveyor line.

[0103] Step S1 (Trigger): The carbon block blocks the photoelectric sensor on the east side, and the state of input point M1.3 of the PLC changes to `True`. The host computer detects this change through periodic PLC status reading (`_pollingTimer_1`, 20ms cycle).

[0104] Step S2 (Image Acquisition): The host computer immediately sends a soft trigger command `MV_CC_SetCommandValue_NET("TriggerSoftware")` to camera 0, and camera 0 acquires an image of the carbon block.

[0105] Step S3 (Image Transmission): The host computer retrieves the image buffer, converts it to JPEG format, and places the (0, JPEG data) tuple into `imageQueue`. The sending thread packages this (index + data length + data) and sends it to the connected Python server.

[0106] Step S4 (AI Processing): The Python server's `receive_image` function receives the data and calls the `process_and_get_info` function. The YOLO model identifies two instances in the image: a "TKA" (container) and a "LW" (block). Python calculates the pixel area and the length and width of the rotated rectangle of the "LW" instance and determines that its bounding box is completely within the bounding box of the "TKA," therefore marking "Judgment: YES" in the information text. Finally, the image with the detection box and mask is encoded as a JPEG.

[0107] Step S5 (Result Feedback and Parsing): The Python server sends the camera index 0, information text length, information text, processed image length, and processed image data back to the host computer in sequence. After receiving the data, the host computer parses the information text and finds that there is a "LW" obsolete block with a status of "YES".

[0108] Step S6 (Laser Ranging and Processing): Simultaneously, the laser rangefinder continuously returns data. In the `JGyes()` method of the host computer, when the photoelectric sensor has a signal, it continuously calculates `TGHeightresul_2` and stores it in the array `TKGD`. When the carbon block leaves and the sensor signal disappears, the algorithm starts processing: Original data: [0,12,25,605,608,610,612,611,609,615,610,608,...] (The first few are data from the rising phase).

[0109] Phase 1: It was found that the data stabilized starting from index 3 (605), so the first 3 data points were removed.

[0110] Phase 2: The median of the stable data is 610, with a calculation range of 595-625. The outlier 615 is removed.

[0111] Phase 3: Take the median of the remaining valid data [605,608,610,612,611,609,610,608], and the final height `TKHHeight` = 610mm.

[0112] Step S7 (Comprehensive Judgment): The host computer integrates two aspects of information: a) Judgment: "LW" defective block is found and is within the valid area; b) Height judgment: 610mm is within the upper limit of the tolerance range (585mm~610mm) and does not exceed the tolerance. According to the preset rule (defective blocks need to be sorted), the carbon block is determined to be a non-conforming product.

[0113] Step S8 (Execute Sorting): The host computer calls the `PLcOut()` method to write `True` to address M0.6 of the PLC. The PLC drives the solenoid valve to open, the cylinder actuates, and the waste block is pushed out of the production line. After the block is pushed out, the photoelectric sensor M1.0 detects that the action is complete, and the host computer resets M0.6 to `False`.

[0114] Step S9 (Data Recording): Throughout the process, all information, including trigger time, image files (raw and processed), detection results, height data, and sorting actions, is recorded in the log file and the corresponding image storage path, and updated and displayed on the UI interface.

[0115] This application successfully integrates advanced AI visual inspection technology, high-precision laser measurement technology and mature industrial control technology to build a high-performance, stable and reliable online inspection and sorting system for anode appearance, which has extremely high practical value and promotion prospects.

[0116] This application may be a method, apparatus, system, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this application.

[0117] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0118] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0119] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing the status information of the computer-readable program instructions. These electronic circuits can execute the computer-readable program instructions to implement various aspects of this application.

[0120] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0121] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0122] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0123] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0124] Note that, unless otherwise explicitly stated, all features disclosed in this specification (including any appended claims, abstract, and drawings) may be replaced by alternative features for achieving the same, equivalent, or similar purpose. Therefore, unless explicitly stated otherwise, each disclosed feature is merely one example of a set of equivalent or similar features. Where used, "further," "preferably," "even further," and "more preferably" are simply starting points for describing another embodiment based on the foregoing embodiments, the combination of which with the foregoing embodiments constitutes the complete configuration of another embodiment. Any combination of several "further," "preferably," "even further," or "more preferably" settings following the same embodiment constitutes yet another embodiment.

[0125] Although this application has been described in detail above with general descriptions and specific embodiments, some modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of this application fall within the scope of protection claimed in this application.

Claims

1. An anode appearance online detection and sorting method based on multi-sensor fusion and artificial intelligence, characterized in that, The application comprises the following steps: running a WPF host computer application, initializing a camera SDK, starting a TCP server, and connecting a PLC controller and a laser range finder; the PLC controller continuously monitors the photoelectric sensor of the production line, and when the photoelectric sensor detects that the material is in place, sends a soft trigger command to the designated camera through the host computer, and the camera collects image data; the host computer acquires camera image data and sends it to the Python server through TCP; the Python server calls the YOLO model for instance segmentation and feature extraction, performs spatial position judgment, generates information text and visualization result graph, and returns them to the host computer; after the material leaves the laser measurement area, the host computer executes a multi-stage processing algorithm on the material height difference data array sent by the laser range finder, and outputs the final height of the material; the host computer parses the information text returned by the Python server to obtain the category and position state of each instance; combined with the category and position state of each instance and the corresponding final height of the material, the quality standard is compared to determine whether the corresponding material needs to be sorted.

2. The multi-sensor fusion and artificial intelligence based online anode appearance inspection and sorting method according to claim 1, characterized in that, After determining whether the corresponding material needs to be sorted, the application further comprises the following steps: if it is determined that the corresponding material is unqualified and needs to be sorted, the host computer writes a True signal to a specific output point of the PLC controller through the S7.Net library; the PLC controller receives the signal and drives the actuator to push the unqualified product out of the production line.

3. The multi-sensor fusion and artificial intelligence based online anode appearance inspection and sorting method according to claim 1, characterized in that, The host computer executes a multi-stage processing algorithm on the material height difference data array sent by the laser range finder to output the final height of the material, specifically comprising the following steps: dynamic caching and rising stage detection: during the detection of the material by the laser range finder, continuously store the height difference value in the array TKGD, find the first stable starting point in the data sequence of the array TKGD that satisfies the condition that the fluctuation of three consecutive points is less than a preset threshold value by using the FindStableStartIndex method, and remove all rising stage data before the point; median-based outlier filtering: calculate the median of the stable stage data after the stable starting point, and based on the median, set a dynamic range, and consider all data outside the dynamic range after the stable starting point as outliers and remove them; robust final value calculation: calculate the median of the valid data sequence obtained after filtering again as the final height of the material.

4. The multi-sensor fusion and artificial intelligence based online anode appearance inspection and sorting method according to claim 3, characterized in that, The comparison of the category and position state of each instance with the corresponding final height of the material with the preset quality standard to determine whether the corresponding material needs to be sorted specifically comprises the following steps: identify all instance bounding boxes of the "TKA" category, and then check whether the instance bounding box of the waste block is completely contained in a certain "TKA" bounding box to obtain a first judgment result; determine whether the final height of the corresponding material exceeds the preset tolerance range to obtain a second judgment result; determine whether the corresponding material needs to be sorted by taking the logical AND or OR of the first judgment result and the second judgment result.

5. A multi-sensor fusion and artificial intelligence based anode appearance online inspection and sorting system performing the method of any one of claims 1 to 4, characterized in that, The application comprises the following steps: The host computer industrial computer, camera array, laser range finder, PLC controller and photoelectric sensor, wherein, The host computer industrial computer is connected with the camera array, laser range finder and PLC controller respectively, and the PLC controller is connected with the photoelectric sensor; The camera array, laser range finder and photoelectric sensor are arranged on a steel frame arranged above a conveying line, and the conveying line is used for conveying carbon blocks; The host computer industrial computer is internally provided with a WPF application, which comprises a camera control module, an image processing and transmission module, a network communication module, a laser ranging communication module, a PLC communication module and a core logic judgment module, wherein, The camera control module encapsulates a camera SDK, which is used for realizing enumeration, parameter configuration, soft trigger acquisition, image cache acquisition and release of the camera; The image processing and transmission module is used for converting the original image data collected by the camera into a format, and sending the image data in the converted format and the camera index to a Python server; The network communication module comprises TCP server and client functions; The laser ranging communication module establishes a connection with the laser range finder through TcpClient, sends a measurement instruction, and asynchronously receives the returned original data; The PLC communication module communicates with the PLC controller by using an S7.Net library, and realizes read and write operations on internal relays of the PLC controller through a `Plc` class object; The core logic judgment module is internally provided with an image result analysis and position judgment algorithm and a laser ranging data multi-stage processing algorithm, which are used for comparing the final height of each instance with a preset quality standard, and judging whether the corresponding material needs to be sorted.

6. The anode appearance online detection and sorting system based on multi-sensor fusion and artificial intelligence according to claim 5, wherein, The image result analysis and position judgment algorithm specifically comprises: From the returned string of the Python server, the detailed information of each instance is parsed using a regular expression, all instance bounding boxes of the "TKA" category are identified, and then it is checked whether the instance bounding box of the waste block is completely contained in a certain "TKA" bounding box to obtain a first judgment result.

7. The anode appearance online detection and sorting system based on multi-sensor fusion and artificial intelligence according to claim 6, wherein, The laser ranging data multi-stage processing algorithm specifically comprises: Dynamic caching and rising stage detection: during the detection of the material by the laser range finder, the height difference value is continuously stored in an array `TKGD`, and a `FindStableStartIndex` method is used to find the first stable starting point in the data sequence of the array `TKGD` that satisfies the condition that the fluctuation of three consecutive points is less than a preset threshold, and all rising stage data before the point is removed; Median-based outlier filtering: the median of the stable stage data after the stable starting point is calculated, and a dynamic range is set based on the median, and all data after the stable starting point that exceeds the dynamic range is regarded as an outlier and is removed. Robust final value calculation: the median of the filtered effective data sequence is calculated again as the final height of the material; determine whether the final height of the corresponding material exceeds the preset tolerance range, to obtain a second determination result.

8. The multi-sensor fusion and artificial intelligence based online anode appearance inspection and sorting system, as claimed in claim 5 wherein, Also includes: Data management and visualization module, for real-time display of camera input / output image, dynamic curve chart of laser measurement data, data table, system log, using LiveCharts control to draw the curve of carbon block height changing with time, and superimposing target line and tolerance line.

9. The multi-sensor fusion and artificial intelligence-based anode appearance online detection and sorting system according to claim 8, characterized in that, The steel frame is erected above the conveying line and includes a stand, a first cross beam, a second cross beam, a third cross beam, a fourth cross beam, a first longitudinal beam, a second longitudinal beam, a third longitudinal beam, a fourth longitudinal beam, and a fifth longitudinal beam. The two ends of the first cross beam, the second cross beam, the first longitudinal beam, and the second longitudinal beam are connected to each other and to the top end of the stand. The third cross beam is arranged parallel below the first cross beam, and the fourth cross beam is arranged parallel below the second cross beam. The third longitudinal beam is arranged parallel below the first longitudinal beam, the fourth longitudinal beam is arranged between the first longitudinal beam and the second longitudinal beam, and the fifth longitudinal beam is arranged parallel below the third longitudinal beam. The camera has multiple cameras, and the third longitudinal beam, the fourth longitudinal beam, and the fifth longitudinal beam are respectively provided with cameras. The laser range finder is arranged on the fourth longitudinal beam, and the photoelectric sensor is arranged on the third cross beam and the fourth cross beam.

10. A computer storage medium having stored thereon a computer program, characterized in that The computer program, when executed by a machine, implements the steps of the method of any one of claims 1 to 4.

Images