Method for context-aware image processing and recognition for detecting surface defects with dynamic allocation of hardware resources

Abstract

FIELD: image processing.SUBSTANCE: method of context-dependent processing and recognition of images for detection of surface defects is characterized in that an information system is used, image capture and buffering is carried out, a balancing unit receives asynchronous streams of digital images with an initial resolution in the range of 10-100 megapixels into memory and distributes them between graphic processors according to the criterion of the smallest number of active connections. A macroframe is generated and a multidimensional vector of the semantic state of the environment and a matrix of spatial density of anomalies are extracted from it. Based on the matrix, coordinates are calculated and a limited number of submatrices of 5-25 pieces are extracted from the original image, each of which has a size of 256 × 256 – 1024 × 1024 pixels. A tensor of local features is extracted from the submatrix, the channels of the tensor of local features are multiplied by the correction coefficients from the multidimensional vector of the semantic state of the environment, and signals that contradict the macrocontext are suppressed. Based on the modulated tensors, coordinates and classes of defects are formed; if the coordinates of defects on intersecting submatrices coincide, spatial confirmation is performed by averaging two maximum values of the defect probability. The final data array is transferred to the output device.EFFECT: accurate analysis of microdefects.1 cl

Description

[0001] Technical field

[0002] The invention relates to the field of information technology and can be applied in the construction industry.

[0003] Technology Level

[0004] A known method for the rapid detection of surface defects in crown caps based on image processing (patent RU 2787072 C1). The method comprises step 1: obtaining an image of the inner surface of the crown cap using an image capture system and processing the obtained image using the Gaussian filtering method; step 2: selecting a central area on the required image using the threshold segmentation method and the method of analyzing the characteristics of an area with connected components and selecting a circular contour on the edge pixels of the central contour of the area using the least squares selection method to determine the central position of the inner surface of the crown cap;step 3: pre-setting the physical radius information of different sections for different types of crown caps according to the coordinates of the central position of the crown cap obtained in step 2, and performing segmentation of the section of the inner surface of the crown cap in combination with the parameters of the pixel ratio of the image; step 4: selecting an image on the desired section of the inner surface according to the result of the image segmentation in step 3, setting a fixed threshold brightness value to complete the image segmentation, and removing the noise section by analyzing the characteristics of the area with connectivity components to perform detection of dark spots on the inner surface of the crown cap;Step 5: extracting an image of the inner and outer annular sections according to the image segmentation result in Step 3, completing the image segmentation by applying a local threshold method, and removing a noise section by a method of analyzing the characteristics of an area with connectivity components to perform bulge detection on the inner and outer annular sections on the inner surface of the crown cap; Step 6: obtaining an edge contour of the outer annular section of the obtained image by a fixed threshold segmentation method, performing a circular selection of pixels of the outer annular contour by a least squares method, and comparing the differences between the outer annular contour section and the selection section to perform rubber absence detection; Step 7: extracting an image of a flash section according to the image segmentation result in Step 3, and performing flash and sealing detection by a fixed threshold segmentation method and a method of analyzing the characteristics of an area;step 8: extracting a connectivity component of the teeth of the skirt based on the image segmentation result in step 3, calculating the number of connectivity components, calculating the internal angle between the connectivity component and the center of the crown cap, and determining whether there are defects on the teeth of the skirt based on the difference in the internal angle of adjacent teeth of the skirt; step 9: obtaining an image of the outer surface of the standard crown cap, performing Gaussian filtering processing on the obtained image, and creating an autonomous feature template according to the feature information of the image contour; step 10: setting a match index with the template, and detecting and positioning the image, which is subject to the detection procedure, by matching the features with the template after processing by the Gaussian filtering method;and step 11: performing an affine transformation on the target image based on the result of matching with the template in step 10, performing threshold segmentation and comparison of the region on the template and target images, respectively, and performing detection of external surface defects by analyzing the characteristics of the region with connectivity components.

[0005] The drawback of this method is that it blindly scans the entire sensor area, resulting in the generation of an excessive number of submatrices (over 100-600 per frame). Attempting to preserve global context with such a large number of local areas causes a quadratic increase in video memory allocation and a hardware buffer overflow on standard computing devices.

[0006] Disclosure of the essence of the invention

[0007] The objective of the invention is to process ultra-high-resolution optical signals.

[0008] The technical result consists of ensuring accurate analysis of microdefects. An additional technical result consists of preventing GPU memory buffer overflows and reducing processing time.

[0009] A method for context-sensitive image processing and recognition for detecting surface defects is characterized by the use of an information system, image capture and buffering by a balancing unit, the reception of asynchronous streams of digital images with a native resolution in the range of 10-100 megapixels into RAM, and their distribution among graphics processors based on the criterion of the smallest number of active connections. Using a computing processor, a macroframe is generated from the original image and a multidimensional vector of the semantic state of the environment and a matrix of spatial anomaly density are extracted from it.Based on the resulting spatial density matrix, the processor calculates coordinates and extracts a limited number of submatrices from the original image, ranging from 5 to 25, each measuring 256x256 to 1024x1024 pixels. A local feature tensor is extracted from each submatrix. The processor then multiplies the local feature tensor channels by correction coefficients derived from the multidimensional semantic state vector of the environment, suppressing signals that contradict the macro context. Based on the modulated tensors, defect coordinates and classes are formed. If defect coordinates on intersecting submatrices match, spatial confirmation is performed by averaging the two maximum defect probability values.The resulting data array is then transferred to an output device, such as a computer monitor, and the processor frees up the RAM buffer for the next asynchronous stream of digital images, maintaining a throughput of at least 3.19 frames per second.

[0010] The load balancer is a hardware and software network gateway (e.g., a dedicated server or microcontroller) equipped with its own network interface, RAM for buffering incoming HTTP requests, and a central processor. It physically acts as a dispatcher: it constantly polls the computing nodes (GPUs) to determine the depth of their task stacks and routes data packets (TCP packets with images) to the GPU with the fewest active connections. This prevents downtime for some processors and hardware failure for others.

[0011] When implementing the claimed method, the image is not randomly divided into multiple fragments, but rather a reduced-size copy of the entire frame is formed. The computing unit analyzes this copy and creates two objects: 1) a "density map" (where defects may physically be located) and 2) a "surface type vector" (macro context).

[0012] Next, the method uses the density map to crop out only a small portion of local zones (micro-tiles). When detecting a defect in a micro-tile, the method applies a hardware "surface type vector" as a filter. This physically blocks those data channels that do not correspond to this surface type (for example, it excludes the search for "ceiling defects" in a "floor" fragment). This solves the problem of context loss without having to load the entire matrix into RAM.

[0013] Hardware Cycle Time Stabilization: Dynamic submatrix allocation (from 5 to 25 per 518x518 frame) and channel modulation enable a processing time of 0.428 s for a single request. Under a peak load of 20 parallel threads per GPU (equal to 16 GB of VRAM), the average processing time does not exceed 4.148 s, and the maximum peak time (99th percentile) is limited to 6 seconds, assuming no hardware classification failures. The lower limit of 5 submatrices is the minimum required optical resolution. When allocating fewer than 5 submatrices, the coverage area of ​​the spatial density matrix becomes insufficient to cover a typical defect cluster in images larger than 10 megapixels. Critical pixels of the defect are lost.

[0014] The upper limit of 25 submatrices is a strict hardware limit for mid-range GPUs (e.g., those with 16 GB of video memory, such as the Nvidia T4). Allocating more than 25 submatrices (518x518 pixels in size) and simultaneously forwarding the semantic state vector causes the video memory allocation limit to be exceeded. This causes either a hardware error or activation of the paging file, increasing processing time from the target of 1-4 seconds to unacceptable values ​​(over 10 seconds).

[0015] Ensuring stable throughput: The physical limitation of the number of processed sub-matrices guarantees the GPU throughput at the level of 3.19-3.23 full recognition cycles per second, regardless of the increase in the number of parallel requests (from 10 to 20).

[0016] Improved defect recognition quality: Blocking conflicting electrical signals based on macro-context vector eliminates the semantic ambiguity of micro-regions, reducing the false positive rate in classifying 340 or more defect types.

[0017] Linear horizontal scaling of computing power: Thanks to independent, stateless stream processing of signals, the introduction of a hardware routing node (a balancer based on the criterion of the least number of active connections) ensures proportional growth of the complex's capacity. Cascading from 1 to 4 GPUs allows for a linear increase in hardware capacity from 20 to 80 parallel threads, reducing the average multi-threaded processing time from 4.1 seconds to 1.0 seconds, respectively.

[0018] Spatial confirmation is a technical operation of hardware optical noise filtering. Because the submatrices are cut with physical overlap (e.g., 50% overlap), the same real-world surface defect is guaranteed to appear in two adjacent submatrices, but at different viewing angles and illumination relative to the frame boundaries.

[0019] This operation confirms the physical presence of a defect. If the computing unit detects an anomaly signal on only one submatrix, while there is no signal in the adjacent (intersecting) zone, the system classifies this as an optical artifact (camera flare, lens distortion, or edge noise). The processor calculates the arithmetic mean of the two maximum probability signal values ​​in the pixel intersection zone of the submatrices. If the average signal exceeds the threshold, the defect is verified. This dramatically reduces the false alarm rate.

[0020] Implementation of the invention

[0021] The method is performed using an information system (for example, a server with a graphics accelerator) connected to an optical recording device and a RAM module, and contains the following sequence of steps:

[0022] Capture and Buffering: The balancing node accepts asynchronous streams of digital images (from 1 to 80 parallel streams) with a native resolution in the range of 10 to 100 megapixels into RAM and distributes them among the GPUs based on the criterion of the smallest number of active connections.

[0023] Macrofeature generation: The computing processor generates a macro frame from the original image and extracts from it a multidimensional vector of the semantic state of the environment (surface type from 25 classes) and a matrix of spatial density of anomalies.

[0024] Dynamic cropping: Based on the received spatial density matrix, the processor calculates the coordinates and selects from the original image a strictly limited number of sub-matrices (micro-tiles) in the amount of 5 to 25 pieces, each of which has a size from 256 × 256 to 1024 × 1024 pixels (for example, 518 × 518 pixels with 50% overlap).

[0025] Hardware modulation of local features: a local feature tensor is extracted from each submatrix. The processor then performs channel cancellation: it multiplies the channels of the local feature tensor by correction coefficients obtained from the multidimensional semantic state vector of the environment. Signals that contradict the macro context are suppressed.

[0026] Classification and Aggregation: Based on the modulated tensors, defect coordinates and classes are formed. If defect coordinates on intersecting submatrices match, spatial confirmation is performed by averaging the two maximum defect probability values, after which the resulting data array is transmitted to the output device. Because the modulation cycle is stateless, the processor immediately frees up the memory buffer for the next stream, maintaining a throughput of at least 3.19 frames per second.

[0027] Measurements under multi-threaded load (from 10 parallel sources) showed a stable throughput of 3.19 hardware cycles per second. Similar information systems using continuous framing or cross-linking fail when attempting to deliver even 0.5 frames per second on a device with similar specifications. Maintaining a throughput of at least 3.19 frames per second demonstrates the effectiveness of the claimed dynamic submatrix allocation algorithm (5-25).