Product nondestructive testing and sorting system based on machine vision

Abstract

The invention relates to the technical field of machine vision and automatic control, and discloses a product nondestructive testing and sorting system based on machine vision, comprising a visual processing unit used for collecting product images, operating an analysis algorithm to determine grade information of products, and sending the grade information to a main control unit; the main control unit is in communication connection with the visual processing unit and is used for receiving the grade information and forwarding the grade information to the sensing and control unit; and the sensing and control unit is in communication connection with the main control unit and comprises a sensing layer, a control layer, an execution layer, a display layer, a network layer and an application layer. According to the system and the method, a three-unit distributed architecture of visual processing, master control, perception and control is adopted, task decoupling is realized, the operation stability is improved, system integration defect quantification and maturity analysis of an LAB color space are comprehensively graded, the detection accuracy is improved, and environment monitoring and accurate sorting are realized.

Description

Technical Field

[0001] This invention relates to the fields of machine vision technology and automated control technology, specifically to a machine vision-based non-destructive testing and sorting system for products. Background Technology

[0002] Machine vision technology is increasingly being used in automated production lines, particularly in product quality inspection and sorting, such as the automated grading of agricultural products. Current technologies typically utilize image acquisition devices to capture product images, then use processors to run image processing algorithms to analyze features such as color, shape, or surface defects, thereby replacing traditional manual sorting and improving production efficiency.

[0003] Conventional inspection and sorting systems often employ centralized control in their structural design. A single main processor must simultaneously handle computationally demanding image algorithm analysis and real-time execution component control tasks. This high degree of task coupling leads to processor resource constraints, impacting the system's real-time response speed and operational stability. Regarding inspection algorithms, existing analysis methods are susceptible to interference from environmental factors. For example, color-based maturity assessments are highly sensitive to changes in ambient lighting, resulting in decreased recognition accuracy.

[0004] Many systems classify products based on a single dimension such as defect size or color. This classification standard is rather crude and cannot provide a comprehensive assessment of product quality. Regarding data interaction and system monitoring, the lack of reliable verification mechanisms in inter-module data communication poses a risk of data misjudgment or loss, leading to sorting errors. Systems also generally lack effective monitoring of operating environment parameters, hindering the troubleshooting of abnormal operating conditions. Therefore, a new technical solution is needed to address the aforementioned problems in existing systems and improve the architectural stability, detection accuracy, and operational reliability of inspection and sorting systems. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a machine vision-based non-destructive testing and sorting system for products. This system solves the problems of high task coupling leading to insufficient operational stability, susceptibility of detection algorithms to ambient light interference and limited grading dimensions, low data communication reliability, and lack of environmental monitoring in existing technologies.

[0006] To achieve the above objectives, the present invention provides a machine vision-based non-destructive testing and sorting system for products, comprising: The vision processing unit is used to acquire product images, run analysis algorithms to determine the product's grade information, and send the grade information to the main control unit; The main control unit is connected in communication with the vision processing unit to receive level information and forward the level information to the perception and control unit. The sensing and control unit communicates with the main control unit to receive grade information and control the execution components to complete the product sorting action based on the grade information.

[0007] In the technical solution provided by this invention, the system adopts a distributed architecture.

[0008] The vision processing unit is dedicated to high-performance image acquisition and complex algorithm analysis; The sensing and control unit is dedicated to acquiring high-real-time sensor data and controlling execution components; The main control unit, as the data communication hub and display center, is responsible for forwarding data between the two and also for displaying local data.

[0009] This architecture decouples high-computation-load tasks from high-real-time control tasks, improving the overall processing efficiency and operational stability of the system.

[0010] In one specific embodiment, the analysis algorithm run by the vision processing unit includes: The target detection algorithm is executed to process the product image, locate the product, and output the region of interest. Execute a defect quantization algorithm to count the area of ​​defective pixels within the region of interest; Execute a maturity analysis algorithm to quantify product maturity within a region of interest; Perform comprehensive product grading to integrate defective pixel area with product maturity to determine product grading information.

[0011] This multi-stage algorithm process first uses a target detection algorithm to quickly locate the product and define the region of interest, providing input for subsequent precise analysis.

[0012] To reduce the interference of ambient lighting changes on color recognition, in one embodiment, the maturity analysis algorithm includes converting the image within the region of interest to the LAB color space. Because the LAB color space separates the luminance and chrominance components, it is more robust to changes in lighting conditions.

[0013] Subsequently, the number of pixels A within the red threshold range was counted in the LAB color space. red This quantifies product maturity. Further, it can be measured by the total pixel area A of the region of interest. total Calculate the maturity percentage R maturity : ; In one embodiment, the defect quantification algorithm specifically includes: Perform grayscale conversion on the region of interest; Erosion morphology operation is performed on the grayscale converted image to eliminate isolated noise points in the image; The Canny edge detection operator is used to process the eroded image to accurately extract the defect contours and generate a binarized image. A region growing algorithm based on eight-neighbor connectivity is used to traverse the binarized image and count the total number of pixels inside all defect contours, thereby obtaining the defect pixel area A. defect .

[0014] The defect area A is quantified. defect Total pixel area A of the region of interest total By comparison, the percentage of defect area R can be further calculated. defect : ; In the aforementioned analysis algorithm, the product comprehensive grading step integrates the defect quantification results with the maturity analysis results. In one embodiment, this step is based on the defect area percentage R. defect and maturity percentage R maturity Through a preset judgment function To determine the final grade information of the product C product For example, the judgment function can be set as follows: ; Among them, R defect R represents the percentage of defect area. maturity C represents the percentage of maturity. product This is the final grade of the product.

[0015] Level 3 represents severe damage, Level 1 represents minor damage and maturity, and Level 2 represents minor damage but immature.

[0016] To ensure the reliability of data transmission between the vision processing unit and the main control unit, in one embodiment, the vision processing unit is specifically used to: process the level information (C... product The data is encapsulated in a data frame including a header and a trailer, and sent to the main control unit via a serial communication interface. Correspondingly, the main control unit is also used to receive and parse the data frame encapsulated with hierarchy information using a state machine-based data receiving mechanism. This mechanism ensures the integrity of the data frame and prevents data corruption.

[0017] In one embodiment, the main control unit is further configured to: update and display the count values ​​of each level of product on a local display device.

[0018] In one embodiment, the internal structure of the sensing and control unit can be designed in layers, including: a sensing layer, a control layer, an execution layer, a display layer, a network layer, and an application layer; wherein, the execution layer includes execution components.

[0019] In one specific embodiment, the sensing layer includes a temperature and humidity sensor, a carbon dioxide sensor, and a photosensor. The sensing layer is preferably connected to the control layer via an IIC bus, and the control layer polls and collects environmental data by accessing different IIC device addresses.

[0020] In one embodiment, the sensing and control unit is also used to enable dual-path concurrent transmission of environmental data: Environmental data is collected periodically by the perception layer; The control layer sends environmental data to the main control unit via a serial interface for display on a local display device. On the other hand, the control layer sends environmental data to the network layer via a serial interface; It is uploaded from the network layer to the application layer via a wireless network.

[0021] This solution combines local monitoring of the system's operating environment with remote monitoring via the cloud.

[0022] In one embodiment, the actuating component includes a servo motor during the sorting operation. Specifically, the control layer is configured to: generate a pulse-width modulation signal via its built-in timer based on the received hierarchy information from the autonomous control unit, and output this signal to the servo motor via a specific pin to precisely control the servo motor to rotate to a predetermined angle, thereby performing the product sorting operation.

[0023] This invention provides a machine vision-based non-destructive testing and sorting system for products. It offers the following advantages: 1. This invention achieves system architecture decoupling by setting up independent vision processing units, main control units, and perception and control units. The vision processing unit is dedicated to image analysis with high computational load, while the perception and control unit is dedicated to sorting operations and environmental acquisition with high real-time requirements. The main control unit serves as the data hub. This distributed structure distributes the processing pressure and improves the overall operating efficiency and stability of the system.

[0024] 2. This invention executes a maturity analysis algorithm in the LAB color space, utilizing the characteristic of separating luminance and chromaticity components in this color space to reduce the interference of ambient light changes on color recognition. The system integrates defect quantification results and maturity analysis results for comprehensive grading, providing a more refined basis for product grade judgment than a single dimension, and improving the accuracy of detection.

[0025] 3. This invention uses data frames containing frame headers and frame tails to encapsulate the analysis results, and the main control unit uses a state machine-based data receiving mechanism to parse them, ensuring the integrity and reliability of data transmission between the visual processing unit and the main control unit. The system concurrently transmits the collected environmental data to the main control unit for local display and to the cloud for reporting at the network layer, realizing dual monitoring functions of local and remote. Attached Figure Description

[0026] Figure 1 This is a system structure circuit block diagram of the present invention; Figure 2 This is a schematic diagram of the wiring of the vision module and the main control unit of the present invention; Figure 3 This is a schematic diagram of the STM32F767IGT6 core board of the present invention; Figure 4 This is a schematic diagram of the STM32F767IGT6 baseboard of the present invention; Figure 5 This is a block diagram of the sensing and control unit system of the present invention; Figure 6 This is a schematic diagram of the sensing layer sensor of the present invention; Figure 7 This is a schematic diagram of the IIC bus wiring of the present invention; Figure 8 This is a schematic diagram of the FPC interface of the present invention; Figure 9 This is a flowchart illustrating the overall program operation of the system according to the present invention; Figure 10 This is a schematic diagram of the visual unit task of the present invention; Figure 11 This is a flowchart of the defect quantization algorithm of the present invention; Figure 12 Flowchart added to the Blinker component of this invention Figure 13 This is a flowchart of the ESP8266 single-loop program of the present invention; Figure 14 This is a logic diagram showing the product classification of the present invention; Figure 15 This is a diagram showing the application layer interface data during system runtime of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions in the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0028] See attached document Figure 1 The product non-destructive testing and sorting system based on machine vision provided by the present invention adopts a distributed design concept. The system may include: a vision processing unit, a main control unit, and a perception and control unit.

[0029] The vision processing unit, main control unit, and perception and control unit exchange data via serial communication.

[0030] The vision processing unit is responsible for acquiring product images and running algorithms for target detection, defect quantification, and maturity analysis. In this embodiment, the vision processing unit uses the OpenMV4H7Plus module. The core processor of this module is an STM32H743 based on the ARM CortexM7 architecture, with a main frequency of up to 480MHz and a computing power of 1027 DMIPS.

[0031] In terms of storage, the vision processing unit has 1MB of built-in RAM and 2MB of FLASH, and is further expanded with 32MB of external RAM and 32MB of external FLASH, for a total storage capacity of 33MB and 34MB respectively. For image acquisition, the module's onboard OV5640 image sensor supports global shutter technology, enabling high-definition image capture of up to 5 megapixels.

[0032] See attached document Figure 2 The vision processing unit and the main control unit are connected via a UART communication interface. Specifically, the TXDP4 port of the vision processing unit is connected to the RXDPB11 port of the main control unit, and the RXDP5 port of the vision processing unit is connected to the TXDPB10 port of the main control unit, and both are grounded together.

[0033] The main control unit is responsible for data communication between the vision processing unit and the perception and control unit, and for generating data on the local display system. In this embodiment, the STM32F767IGT6 chip is selected as the core main control unit. This chip is based on the ARM Cortex M7 core, has a main frequency of up to 216MHz, and is equipped with 1024K of FLASH and 512K of SRAM.

[0034] The main control unit is connected to a 7-inch RGB LCD with a resolution of 1024*600 to achieve data display. (See attached diagram.) Figure 3 and attached Figure 4 The development board used for the main control unit consists of a core board and a baseboard, which are connected via a B2B interface.

[0035] The sensing and control unit is responsible for sensor data acquisition, control output, and cloud platform data display. (See attached document.) Figure 5 The perception and control unit, as a subsystem, can be further divided into the perception layer, control layer, execution layer, display layer, network layer, and application layer.

[0036] The perception layer is responsible for collecting environmental data. (See attached diagram) Figure 6 The sensing layer includes: a photosensitive sensor GY302 with BH1750FVI-TR as its core component; a carbon dioxide sensor with SGP30-2.5k as its core component; and a temperature and humidity sensor SHT30-DIS-B10KS.

[0037] See attached document Figure 7 The temperature and humidity sensors, carbon dioxide sensors, and photosensors within the sensing layer all employ the IIC communication protocol. This embodiment uses an IIC bus mode, meaning the three sensors are connected to the same set of IIC bus pins in the control layer. The control layer identifies the address codes of different IIC devices (0x58 for SGP30, 0x46 for BH1750FVI, and 0x44 for SHT30) and sequentially acquires data from each sensor.

[0038] The control layer uses the STM32F103C8T6 as the main control chip. This microcontroller is based on the ARM 32-bit Cortex-M3 CPU core, has a 72MHz operating frequency, and is equipped with 8KB of flash program memory and 20KB of SRAM.

[0039] Referring to Table 1, USART1 in the control layer is used for data transmission and reception with the main control unit; USART2 is used for data transmission and reception with the ESP8266 in the network layer; I2C2 is used to connect to the IIC bus in the sensing layer; and I2C1 is used to connect to the OLED in the display layer. In addition, the timer PWM channel is used to control the corresponding devices in the execution layer.

[0040] The execution layer integrates a cooling fan, a passive buzzer, dual servos, and auxiliary lights to execute specific control commands. The display layer uses a 0.42-inch OLED display and achieves data interaction with the control layer via the IIC communication protocol.

[0041] Table 1: Wiring Table for STM32F103C8T6 Control Layer of Sensing and Control Unit

[0042] See attached document Figure 8 The output pins of the execution layer are integrated into a 20-pin FPC cable interface.

[0043] The network layer uses the ESP8266 (ESP-05) WiFi module. The ESP8266 WiFi module encapsulates and parses data according to the MQTT communication protocol, enabling data interaction with the application layer. The application layer utilizes the mobile app Blinker to provide users with a customizable and editable user interface and visualized data display.

[0044] See attached document Figure 9 This invention provides a working method for a machine vision-based non-destructive testing and sorting system for products, comprising the following steps: S1. The vision processing unit performs image acquisition and analysis: it acquires product images and sequentially runs the target detection algorithm, defect quantification algorithm, and maturity analysis algorithm to determine the product's grade information. S2. The vision processing unit encapsulates and sends the data frame of the analysis results: the determined product grade information is sent to the main control unit via the serial communication interface. S3. The main control unit receives, parses, and distributes data: it receives and parses the grade information from the vision processing unit, displays the number of products of each grade on the local 7-inch LCD screen, and sends the latest grading results to the perception and control unit via the serial communication interface. S4. The sensing and control unit receives and executes instructions: The control layer receives the hierarchical results from the main control unit and controls the servo motor in the execution layer to perform the corresponding product sorting actions based on the received hierarchical results. S5. The sensing and control unit collects and transmits environmental data bidirectionally: The sensing layer collects light intensity data, carbon dioxide concentration, and temperature and humidity data in the environment. The control layer sends the collected environmental data to the main control unit via USART and then to the network layer. The network layer uses the MQTT communication protocol to upload the data to the application layer for cloud display.

[0045] The steps described above in the embodiments of the present invention will be explained in detail below.

[0046] In step S1, the image acquisition and analysis performed by the vision processing unit are explained.

[0047] See attached document Figure 10 This process can specifically include the following sub-steps: S11. Perform target detection: The vision processing unit drives its photosensitive element to capture product images. The system calls a pre-trained model file deployed in internal storage to process the captured images, perform inference on the images, identify and locate the positions of product targets in the images, and output a region of interest (ROI) containing the product target.

[0048] S12. Execution Defect Quantification: This process is performed within the Region of Interest (ROI) defined in S11. (See Appendix) Figure 11 First, the region of interest is cropped and converted into a grayscale image. Then, an erosion morphological operation is performed on the grayscale image to eliminate isolated noise points. Subsequently, the Canny edge detection operator is used to process the eroded image to generate a binary image containing the defect contour. Finally, a region growing algorithm based on eight-neighborhood connectivity is used to traverse the binary image, count the total number of pixels inside the defect contour, and obtain the quantized area of ​​the defect.

[0049] S13. Perform maturity analysis: This process is also performed within the region of interest (ROI) determined in S11. To reduce the impact of changes in ambient lighting, this embodiment performs the analysis in the LAB color space. By calling the find_colour function built into the visual processing unit and setting a red color threshold range, the number of pixels within the ROI that meet the preset red threshold range is counted, and their proportion in the total pixels of that region is calculated to evaluate product maturity.

[0050] The code for the find_colour function is as follows: def find_colour(img, ROI): red_sum=0 for blob in img.find_blobs(red_threshold, roi = ROI, pixels_threshold=2, area_threshold=2, merge=True): red_sum += blob.pixels() return red_sum S14. Perform comprehensive product grading: Integrate the results of defect quantification and maturity analysis to achieve the final product grading. (See appendix) Figure 14 Let R defect R is the percentage of the defect area to the total product area calculated in sub-step S12. maturity The percentage of red area calculated in sub-step S13, product grade C product Based on the following judgment function Sure: ; Among them, product grade C product A grade of 3 represents a severely damaged product; grade 1 represents a less damaged and mature product; and grade 2 represents a less damaged but immature product.

[0051] To ensure the stable and continuous operation of the aforementioned algorithms on the vision processing unit and to avoid memory allocation failures due to memory fragmentation, this embodiment calls the micropython.alloc_emergency_exception_buf() function during the program initialization phase to pre-allocate an emergency exception buffer, thereby ensuring the stable operation of the system.

[0052] In step S2, the encapsulation and transmission of data frames representing the analysis results from the vision processing unit are explained.

[0053] This step may specifically include the following sub-steps: S21. Perform data frame encapsulation: The vision processing unit encapsulates the determined product grade C. product This core data is encapsulated within a predefined data frame. To ensure data transmission integrity, the data frame structure includes a frame header, data, and a frame trailer, used to identify the beginning and end of a data frame.

[0054] S22. Execute data frame transmission: After encapsulation, the vision processing unit sends the data frame byte by byte to the main control unit through its serial communication interface in the order of frame header, data, and frame tail.

[0055] In step S3, the data reception, parsing, and distribution of the main control unit are explained.

[0056] This step may specifically include the following sub-steps: S31. Receiving and parsing data frames: The main control unit monitors the data stream in real time through its serial interface connected to the vision processing unit. This embodiment employs a state machine-based data reception mechanism (such as the ProcessVisionData function) to ensure the integrity of the data frames. This mechanism switches between states such as idle, frame header found, and data reception through a state variable. When the preset frame header byte is detected, data reception begins; when the frame tail byte is detected, it indicates that a frame of data has been received.

[0057] The code for the ProcessVisionData function is as follows: void ProcessVisionData(uint8_t data) { static uint8_t rxBuffer

[128] ; static int bufferIndex = 0; static enum {IDLE, HEADER_FOUND, DATA_RECEIVING} rxState = IDLE; if (rxState == IDLE && data == 0xFF) { rxState = HEADER_FOUND; bufferIndex = 0; } else if (rxState == HEADER_FOUND) { rxBuffer[bufferIndex++] = data; if (bufferIndex >= 1) / / Assuming the data length is 1 byte { rxState = DATA_RECEIVING; } } else if (rxState == DATA_RECEIVING) { rxBuffer[bufferIndex++] = data; if (data == 0xFE) { / / Detected frame end, parse the complete data packet ParseVisionDataFrame(rxBuffer, bufferIndex); rxState = IDLE; } } } void ParseVisionDataFrame(uint8_t* frame, int length) { / / Analyze the data frames sent by the vision processing unit to extract level information and sensor data. ExtractFruitClassData(frame, length); ExtractSensorData(frame, length); gSystemState.updateDisplay = 1; / / Update the display flag ForwardDataToControlUnit(frame, length); / / Forward data to the perception control unit } S32. Data Parsing: After receiving a complete data frame, the main control unit processes the data in the receive buffer and extracts the product grade C determined by the vision processing unit. product information.

[0058] S33. Execute local data display: The main control unit maintains the count values ​​for each product level. After parsing to obtain the new product level C... product Then, the corresponding level's count value is updated. A separate display task will refresh and display the updated product count values ​​for each level (as well as the environmental parameters received in S5 later) on the local display screen.

[0059] S34. Forwarding of sorting instructions: The main control unit parses and obtains product grade C. product Then, this level information is used as a sorting instruction. The main control unit calls a data forwarding function (such as the ForwardDataToControlUnit function) to send the sorting instruction to the sensing and control unit through its other independent serial communication interface.

[0060] In step S4, the sensing and control unit receives and executes commands.

[0061] This step may specifically include the following sub-steps: S41. Receiving Sorting Instructions: The control layer receives sorting instructions sent from the main control unit through its pre-configured serial communication interface USART1. This receiving process is implemented through a serial port service interrupt function to ensure the real-time nature of instruction reception.

[0062] The specific code for the segment function in the serial port service is as follows: / * Serial port service interrupt function in Serial.c * / #include "stm32f10x.h" #include<stdio.h> #include<stdarg.h> extern int counters[3]; / / external reference ...... void USART1_IRQHandler(void) / / Serial port interrupt function { / / Read the level information from the main control unit if (USART_GetITStatus(USART1, USART_IT_RXNE) == SET) { uint8_t newone = USART_ReceiveData(USART1); / / Storage if(newone >= 1 && newone <= 3) { / / Send to the ESP8266 IoT module at the network layer usart2_SendByte(newone); / / Update data counters[newone-1]++; } USART_ClearITPendingBit(USART1,USART_IT_RXNE); } } S42. Perform sorting action: The control layer receives and confirms product grade C. product Following the instruction, the information at that level is converted into a control signal for a specific servo motor in the execution layer according to preset logic. The control layer uses its timer to generate a pulse width modulation signal, which is then output to the servo motor in the execution layer. The rotation of the servo motor drives the corresponding sorting mechanism to move, guiding the current product to the corresponding collection area, thus completing the physical sorting.

[0063] In step S5, the environmental data acquisition and bidirectional transmission of the sensing and control unit are explained.

[0064] This step may specifically include the following sub-steps: S51. Acquisition of Environmental Data: The control layer interacts with the sensing layer connected to the IIC bus through its IIC communication interface. The control layer uses a time-division multiplexing approach, sending read commands sequentially to each sensor on the IIC bus to acquire environmental data.

[0065] S52. Two-way transmission of execution environment data: After obtaining the latest environment data, the control layer transmits the data concurrently through two independent paths: Path 1 (Transmission to the main control unit): The control layer sends the collected environmental data to the main control unit through its serial communication interface, so that the main control unit can execute the local data display logic in S33.

[0066] Path Two (Transmission to Network Layer): The control layer sends the same environmental data to the WIFI module in the network layer via its serial communication interface. The WIFI module encapsulates the data according to the MQTT communication protocol and publishes it to a predetermined topic in the application layer via the WIFI network, enabling remote cloud monitoring of the environmental data.

[0067] See attached document Figure 10 This invention provides a core algorithm for a machine vision-based non-destructive testing and sorting system for products. This algorithm is primarily executed in the vision processing unit and specifically includes: Target detection algorithm: The pre-trained FOMO MobileNetV2 0.1 model is used to infer the acquired images to identify and locate product targets and output the region of interest; Defect quantification algorithm: Within the region of interest, image grayscale conversion, erosion operation, Canny edge detection, and eight-neighborhood-based region growing algorithm are performed sequentially to count the defect pixel area; Maturity analysis algorithm: Within the region of interest, based on the LAB color space, the findcolour function is used to calculate the proportion of red pixels to quantify product maturity; Product comprehensive grading: Integrates defect quantification results and maturity analysis results, and determines the final grade of the product based on preset thresholds and a grading judgment function.

[0068] The implementation principles of the various algorithms described in the embodiments of the present invention will be explained in detail below.

[0069] The implementation principle of the object detection algorithm is explained. The FOMO MobileNetV2 0.1 model used in this embodiment was obtained by training on the Edge Impulse cloud platform. The training process used a dataset containing 675 images of Fuji apple products, performed 60 rounds of iteration, and applied int8 quantization to the model after training, finally generating the deployable model file trained.tflite.

[0070] The trained.tflite model file is deployed within the vision processing unit. During object detection, the vision processing unit loads this model file and calls the model to perform inference on the acquired real-time images. This inference operation is used to quickly identify and locate product targets in the image, ultimately outputting a region of interest (ROI) that defines the location of the product. The ROI will then be used as the processing object for subsequent defect quantification and maturity analysis algorithms.

[0071] The implementation principle of the defect quantification algorithm is explained. This algorithm is executed within the region of interest (ROI) determined by the target detection algorithm, and its purpose is to accurately quantify the defect area on the product surface.

[0072] See attached document Figure 11 The specific implementation of this algorithm may include: The region of interest (ROI) output by the object detection algorithm is cropped and converted into a grayscale image.

[0073] Perform an erosion morphological operation on the grayscale image. This operation uses a structuring element (kernel) to slide across the image, assigning the minimum value in the neighborhood of a pixel to the center pixel. The effect is to eliminate isolated noise points in the image caused by lighting or sensors, preventing these noise points from being misjudged as defects in subsequent processing.

[0074] The image after erosion is processed using the Canny edge detection operator. Canny edge detection, through image gradient calculation, nonmaximum suppression, and double thresholding, can accurately extract the true contours of surface defects and generate a binary image containing only the defect contours.

[0075] Defect area statistics are performed. A region growing algorithm based on eight-neighborhood connectivity is used on the generated binarized image. This algorithm traverses the image and counts the total number of pixels located inside the defect contours detected by the Canny operator. This total number is the quantized area of ​​the defect, denoted as A. defect .

[0076] To facilitate subsequent classification, the defect area A is quantified. defect The total pixel area A of the region of interest (ROI) total By comparison, the percentage of defect area R is calculated. defect The calculation formula is as follows: ; This defect area percentage A defect It will serve as one of the important input parameters for comprehensive product grading.

[0077] The implementation principle of the maturity analysis algorithm is explained. This algorithm is also executed within the region of interest (ROI) determined by the target detection algorithm. Its purpose is to objectively and quantitatively evaluate the maturity of the product.

[0078] To reduce the interference of changes in ambient light intensity on the accuracy of color recognition, this embodiment does not use the conventional RGB color space, but instead converts the image within the region of interest (ROI) to the LAB color space. The LAB color space separates the luminance (L) component from the chrominance (a,b) components, making the color information more robust to changes in illumination.

[0079] In the LAB color space, the region of interest (ROI) is processed by calling the `find_colour` function built into the visual processing unit. This function is pre-configured with a set of color thresholds for defining red. When the function runs, it iterates through all pixels within the ROI and counts the total number of pixels whose color values ​​fall within the preset red threshold range, denoted as A. red .

[0080] The total number of red pixels A obtained from the statistics red The total pixel area A of the region of interest (ROI) total By comparing the percentage of red area, the maturity index R is calculated. maturity The calculation formula is as follows: ; This maturity percentage R maturity This will serve as another important input parameter for comprehensive product grading.

[0081] The implementation of the comprehensive product grading logic is explained, which integrates the analysis results of the aforementioned algorithms to determine the final grade of the product.

[0082] The system obtains the defect area percentage R calculated by the defect quantification algorithm. defect The maturity percentage R calculated in the maturity analysis algorithm maturity These two values ​​are used as input parameters for comprehensive grading.

[0083] The system uses a preset judgment function. To determine the final grade C of the product product The specific logic of this judgment function is as follows: ; Among them, R defect R represents the percentage of defect area. maturity C represents the percentage of maturity. product This is the final grade of the product.

[0084] In this embodiment, the defect area threshold is 10%, and the maturity threshold is 70%. Based on this logic, product grade C... product Level 3 represents severely damaged products; Level 1 represents less damaged and mature products; Level 2 represents less damaged but immature products. The final level is C. product It will be output for use in subsequent data frame encapsulation and transmission.

[0085] To ensure the algorithm runs stably and continuously on the vision processing unit and avoids memory allocation failures due to memory fragmentation caused by long-term operation, this embodiment calls the `micropython.alloc_emergency_exception_buf()` function during the program initialization phase. This function pre-allocates an emergency exception buffer, thereby ensuring the stability of the algorithm's execution.

[0086] See attached document Figure 5 Appendix Figure 12 and attached Figure 13 This invention provides the implementation logic of the software programs of each unit in a machine vision-based product non-destructive testing and sorting system, specifically including: The software program implementation logic of the vision processing unit is as follows: After initializing the photosensitive element and emergency exception buffer, the program enters a main loop. In each loop, the program drives the photosensitive element to capture images, calls the deployed trained.tflite model file for inference to obtain the region of interest of the product target, then executes defect quantification and maturity analysis algorithms within the region, determines the product level based on the analysis results, and finally encapsulates the level information into a data frame of a predetermined format and sends it to the main control unit through the serial communication interface.

[0087] The main control unit's software implementation logic is as follows: After initializing the LCD screen and various serial communication interfaces, the program processes the data stream from the vision processing unit through a state machine-based data receiving mechanism in its main loop to ensure the integrity and correctness of the data frames. After parsing valid product level information, the program updates the local product count value, sets the display update flag, and forwards the level information to the perception and control unit through another serial interface. An independent display task monitors the update flag and refreshes the latest count value and environmental data to the LCD screen.

[0088] The software implementation logic of the control layer of the perception and control unit mainly consists of interrupt services and a main loop task. The program receives interrupts via a serial interface to obtain product level instructions sent by the main control unit in real time, and generates corresponding PWM signals through a timer to drive the sorting servo motor according to these instructions. Simultaneously, the program periodically polls the environmental sensors in the perception layer via the IIC bus to collect data, and sends the collected data to the main control unit for local display and to the network layer for remote reporting via two independent serial interfaces.

[0089] The software implementation logic of the network layer of the perception and control unit is shown in the appendix. Figure 5 and attached Figure 13After initializing the Wi-Fi connection and MQTT client, the program continuously listens for serial data from the control layer in its main loop. Once environmental data is received, the program immediately encapsulates it according to the MQTT protocol format and publishes it to the MQTT topic specified by the application layer via the Wi-Fi network.

[0090] Application layer software implementation logic: The application layer interface relies on the Blinker platform for visual design and editing. It utilizes intuitive and convenient interactive operations to configure device functions and build interface components. The complete workflow for device configuration and component addition on the Blinker platform is attached. Figure 12 As shown.

[0091] The Blinker program acts as an MQTT client, subscribing to environmental data topics published by the network layer. When a new data message is received, the program parses the message and updates the contained environmental parameters to the visual user interface of the mobile device, thereby enabling remote monitoring of the field environment.

[0092] To verify the actual performance of the machine vision-based non-destructive testing and sorting system provided in this embodiment of the invention, a physical system was built and its performance was tested and analyzed. In the neural network recognition performance test, the trained.tflite model file trained and exported from the Edge Impulse platform was deployed to the vision processing unit. During system operation, the vision processing unit acquired images and called the model for inference. Test results show that the neural network model can stably, quickly, and accurately identify the location and quantity of products within the field of view, and can complete multiple recognition tasks within 1 second, providing a reliable target area input for subsequent defect quantification and maturity analysis.

[0093] In the performance tests of defect quantification and maturity analysis, the product location information output by the aforementioned target detection was used as the input to the defect quantification algorithm and the maturity analysis algorithm. Test results show that the defect quantification algorithm can stably perform area statistics on surface defects of products with different shapes; the analysis results of the maturity analysis algorithm are consistent with industry understanding and can effectively assess product quality.

[0094] In the data communication test, the serial communication link between the vision processing unit, the main control unit, and the perception and control unit was verified. During the test, the product level information determined by the vision processing unit was smoothly and accurately forwarded to the perception and control unit via the main control unit, without any data loss or errors, ensuring the accuracy and efficiency of data transmission between the various units of the system.

[0095] In performance testing of the sensing and control unit, its control layer data acquisition program ran normally, stably reading data from various sensors in the sensing layer via the IIC bus. Regarding output control, the servo motors, auxiliary lights, and other actuators all responded promptly and correctly to control commands, verifying the reliability of its sensing and control functions. (See attached document) Figure 15 The real-time updated environmental data in the figure shows that the data reporting function of the network layer and the remote monitoring function of the application layer are operating normally.

[0096] In the performance test of the main control unit, the data display and update function of the LCD screen operated normally, and was able to present the system status information such as the count values ​​and environmental parameters of each level of product in a timely and clear manner. It could also provide feedback for abnormal situations, realizing the visual monitoring of the system operation status.

Claims

1. A machine vision-based non-destructive product inspection and sorting system, characterized in that, include: The vision processing unit is used to acquire product images, run analysis algorithms to determine the product's grade information, and send the grade information to the main control unit; The main control unit is communicatively connected to the vision processing unit and is used to receive the level information and forward the level information to the perception and control unit; The sensing and control unit is communicatively connected to the main control unit and is used to receive the level information and control the execution components to complete the product sorting action based on the level information.

2. The machine vision-based non-destructive testing and sorting system for products according to claim 1, characterized in that, The analysis algorithms run by the visual processing unit include: An object detection algorithm is executed to process the product image to locate the product and output the region of interest; A defect quantization algorithm is executed to count the defect pixel area within the region of interest; A maturity analysis algorithm is executed to quantify the product maturity within the region of interest. Perform a comprehensive product grading process to integrate the defective pixel area with the product maturity level to determine the product's grading information.

3. The machine vision-based non-destructive testing and sorting system for products according to claim 2, characterized in that, The defect quantification algorithm specifically includes: Perform grayscale conversion on the region of interest; Perform erosion morphology operations on the grayscale-converted image; The Canny edge detection operator is used to extract defect contours to generate a binarized image; The area of ​​defective pixels in the binarized image is counted using a region growing algorithm based on eight-neighbor connectivity.

4. The machine vision-based non-destructive testing and sorting system for products according to claim 2, characterized in that, The maturity analysis algorithm includes: converting the image within the region of interest to the LAB color space, and counting the number of pixels within a preset color threshold range in the LAB color space to quantify the product maturity.

5. The machine vision-based non-destructive testing and sorting system for products according to claim 1, characterized in that, The visual processing unit is specifically used to: encapsulate the level information in a data frame including a frame header and a frame tail, and send it to the main control unit through a serial communication interface.

6. The machine vision-based non-destructive testing and sorting system for products according to claim 1, characterized in that, The main control unit is also used for: A state machine-based data receiving mechanism is used to receive and parse data frames encapsulated with the level information. Update and display the count values ​​for each product level on the local display device.

7. The machine vision-based non-destructive testing and sorting system for products according to claim 1, characterized in that, The perception and control unit includes: a perception layer, a control layer, an execution layer, a display layer, a network layer, and an application layer; the execution layer includes the execution component.

8. The machine vision-based non-destructive testing and sorting system for products according to claim 7, characterized in that, The sensing and control unit is also used for: The sensing layer periodically collects environmental data; The control layer sends the environmental data to the main control unit for local display and also sends it to the network layer; The data is uploaded from the network layer to the application layer via a wireless network.

9. The machine vision-based non-destructive testing and sorting system for products according to claim 8, characterized in that, The sensing layer includes a temperature and humidity sensor, a carbon dioxide sensor, and a photosensor. The sensing layer is connected to the control layer via an IIC bus so that the control layer can collect the environmental data.

10. The machine vision-based non-destructive testing and sorting system for products according to claim 7, characterized in that, The actuator includes a servo motor; the control layer is specifically used to control the servo motor to perform the product sorting action by generating a PWM signal.

Images