Intelligent machine vision detection method and system based on image processing and storage medium

Abstract

The invention relates to the technical field of image processing, and discloses an intelligent machine vision detection method and system based on image processing, and a storage medium. The method comprises the steps of performing candidate region generation and geometric constraint optimization processing on fusion features through a cascade bidirectional region suggestion network to obtain a target candidate region, and performing instance segmentation processing on the target candidate region through a graph convolutional network and a boundary dynamic adjustment mechanism to obtain a segmentation result; performing feature extraction processing on the segmentation result through a space-time joint attention network to obtain space-time context features; performing multi-target tracking processing on the spatio-temporal context features through depth correlation measurement and motion mode matching to obtain a trajectory prediction result; and performing abnormal mode modeling processing on the trajectory prediction result through hierarchical contrast learning and multi-modal feature fusion to obtain abnormal features, and generating a visual inspection report according to the abnormal features. According to the invention, the accuracy of intelligent machine vision detection based on image processing is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an intelligent machine vision detection method, system and storage medium based on image processing. Background Art

[0002] In industrial production, intelligent machine vision inspection technology is widely used in product quality inspection, industrial robot control, and production process monitoring. Existing machine vision inspection methods mainly include two categories: detection methods based on traditional image processing and detection methods based on deep learning. Traditional image processing methods achieve target detection and defect recognition through steps such as image preprocessing, feature extraction, and pattern recognition, but they have poor adaptability to complex scenes and dynamic changes. Although deep learning-based detection methods have strong feature learning capabilities, they often require a large amount of labeled data for training, and have high requirements for real-time and computing resources. In actual industrial applications, it is also necessary to consider multiple links such as target tracking, anomaly detection, and risk assessment at the same time. Existing methods often separate these tasks and lack systematicity and coherence.

[0003] The existing technology has the following shortcomings: First, in the image preprocessing stage, a single processing method is difficult to adapt to various interference factors in industrial scenes, such as illumination changes, noise and occlusion; second, the multi-scale and multi-modal information is not fully utilized in the feature extraction process, resulting in insufficient feature expression ability; third, the target detection and tracking process lacks modeling of spatiotemporal context information, which affects the system's ability to understand dynamic scenes; finally, the anomaly detection and risk assessment links lack a hierarchical analysis framework, making it difficult to achieve accurate anomaly positioning and risk quantification. Summary of the invention

[0004] The present application provides an image processing-based intelligent machine vision detection method, system and storage medium for improving the accuracy of image processing-based intelligent machine vision detection.

[0005] In a first aspect, the present application provides an intelligent machine vision detection method based on image processing, the intelligent machine vision detection method based on image processing comprising: performing multi-scale decomposition and adaptive threshold segmentation processing on the original image to obtain a segmented image, and performing multi-layer cascade filtering and nonlinear weighted fusion processing on the segmented image to obtain a preprocessed image; performing complementary feature extraction processing on the preprocessed image through a parallel multi-branch heterogeneous neural network to obtain multi-dimensional features, and performing feature fusion processing on the multi-dimensional features through a hierarchical adaptive attention mechanism and cross-scale feature recalibration to obtain fused features; performing cascade bidirectional region filtering on the fused features to obtain fused features. The domain proposal network performs candidate region generation and geometric constraint optimization processing to obtain the target candidate region, and performs instance segmentation processing on the target candidate region through a graph convolutional network and a boundary dynamic adjustment mechanism to obtain a segmentation result; the segmentation result is subjected to feature extraction processing through a spatiotemporal joint attention network to obtain spatiotemporal context features; the spatiotemporal context features are subjected to multi-target tracking processing through deep association measurement and motion pattern matching to obtain a trajectory prediction result; the trajectory prediction result is subjected to abnormal pattern modeling processing through hierarchical contrast learning and multimodal feature fusion to obtain abnormal features, and a visual detection report is generated based on the abnormal features.

[0006] In a second aspect, the present application provides an intelligent machine vision detection device based on image processing, and the intelligent machine vision detection device based on image processing includes:

[0007] The segmentation module is used to perform multi-scale decomposition and adaptive threshold segmentation processing on the original image to obtain a segmented image, and perform multi-layer cascade filtering and nonlinear weighted fusion processing on the segmented image to obtain a pre-processed image;

[0008] A fusion module is used to perform complementary feature extraction processing on the pre-processed image through a parallel multi-branch heterogeneous neural network to obtain multi-dimensional features, and perform feature fusion processing on the multi-dimensional features through a hierarchical adaptive attention mechanism and cross-scale feature recalibration to obtain fused features;

[0009] An optimization module, used to generate candidate regions and perform geometric constraint optimization processing on the fused features through a cascaded bidirectional region proposal network to obtain a target candidate region, and perform instance segmentation processing on the target candidate region through a graph convolution network and a boundary dynamic adjustment mechanism to obtain a segmentation result;

[0010] An extraction module, used for performing feature extraction processing on the segmentation result through a spatiotemporal joint attention network to obtain spatiotemporal context features;

[0011] A tracking module, used to perform multi-target tracking processing on the spatiotemporal context features through deep association measurement and motion pattern matching to obtain trajectory prediction results;

[0012] The generation module is used to perform abnormal pattern modeling processing on the trajectory prediction result through hierarchical contrast learning and multimodal feature fusion to obtain abnormal features, and generate a visual detection report based on the abnormal features.

[0013] A third aspect of the present application provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the above-mentioned intelligent machine vision detection method based on image processing.

[0014] In the technical solution provided in this application, by adopting a combined processing method of multi-scale decomposition and adaptive threshold segmentation, combined with multi-layer cascade filtering and nonlinear weighted fusion, the robustness and adaptability of image preprocessing can be effectively improved, ensuring that high-quality preprocessed images are obtained in complex industrial environments. The design of parallel multi-branch heterogeneous neural networks fully utilizes the advantages of different network structures, and cooperates with the hierarchical adaptive attention mechanism and cross-scale feature recalibration to achieve complementary extraction and effective fusion of multi-dimensional features, significantly enhancing the expression ability of features. The introduction of the cascaded bidirectional region proposal network improves the accuracy of target detection through candidate region generation and geometric constraint optimization. The combination of graph convolutional network and boundary dynamic adjustment mechanism further optimizes the accuracy of instance segmentation, and at the same time, the spatiotemporal The application of joint attention network realizes the effective modeling of temporal information and enhances the spatiotemporal consistency of feature extraction. The multi-target tracking strategy through deep correlation measurement and motion pattern matching not only improves the accuracy of target tracking, but also effectively handles the multi-target interaction problem in complex scenarios. Finally, the anomaly detection framework of hierarchical contrast learning and multimodal feature fusion realizes a full range of anomaly analysis from low-level physical features to high-level semantic features, and provides intuitive and explainable detection results through structured visual inspection reports. The overall solution constructs a complete intelligent machine vision inspection process through the organic combination of multiple innovative technical features. While improving the detection accuracy, it also ensures the real-time and reliability of the system, which is particularly suitable for the actual application needs of industrial sites. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0016] Figure 1 A schematic diagram of an embodiment of an intelligent machine vision detection method based on image processing in an embodiment of the present application;

[0017] Figure 2 This is a schematic diagram of an embodiment of an intelligent machine vision detection device based on image processing in an embodiment of the present application. DETAILED DESCRIPTION

[0018] The present application embodiment provides a kind of intelligent machine vision detection method, system and storage medium based on image processing. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or precedence. It should be understood that the data used in this way can be interchangeable in appropriate circumstances, so that the embodiments described here can be implemented in an order other than the content illustrated or described here. In addition, the term "including" or "having" and any variation thereof are intended to cover non-exclusive inclusions, for example, the process, method, system, product or equipment comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0019] For ease of understanding, the specific process of the embodiment of the present application is described below. Figure 1 , an embodiment of the intelligent machine vision detection method based on image processing in the embodiment of the present application includes:

[0020] Step S101, performing multi-scale decomposition and adaptive threshold segmentation processing on the original image to obtain a segmented image, and performing multi-layer cascade filtering and nonlinear weighted fusion processing on the segmented image to obtain a preprocessed image;

[0021] Step S102, performing complementary feature extraction processing on the preprocessed image through a parallel multi-branch heterogeneous neural network to obtain multi-dimensional features, and performing feature fusion processing on the multi-dimensional features through a hierarchical adaptive attention mechanism and cross-scale feature recalibration to obtain fused features;

[0022] Step S103: Generate candidate regions and perform geometric constraint optimization processing on the fused features through a cascaded bidirectional region proposal network to obtain target candidate regions, and perform instance segmentation processing on the target candidate regions through a graph convolutional network and a boundary dynamic adjustment mechanism to obtain a segmentation result;

[0023] Step S104: performing feature extraction processing on the segmentation result through the spatiotemporal joint attention network to obtain spatiotemporal context features;

[0024] Step S105, performing multi-target tracking processing on the spatiotemporal context features through deep association measurement and motion pattern matching to obtain trajectory prediction results;

[0025] Step S106: The trajectory prediction results are processed by abnormal pattern modeling through hierarchical contrast learning and multimodal feature fusion to obtain abnormal features, and a visual inspection report is generated based on the abnormal features.

[0026] It is understandable that the execution subject of the present application may be an intelligent machine vision detection device based on image processing, or a terminal or a server, which is not limited here. The present application embodiment is described by taking the server as the execution subject as an example.

[0027] Specifically, the intelligent machine vision detection method based on image processing first performs multi-scale decomposition and adaptive threshold segmentation on the input original image. The multi-scale decomposition process adopts a pyramid structure, and the original image is scaled by setting the scale factors of 0.5, 0.75 and 1.0 to generate three sub-images with different resolutions. For each sub-image, the entropy distribution map of the local area is calculated, the window size is set to 7×7 pixels, and the grayscale distribution probability of each position is counted by sliding the window to obtain the entropy map. Based on the entropy map, the iterative minimum cross entropy algorithm is used for adaptive threshold selection. The initial threshold is set to the average grayscale value of the image. After multiple rounds of iterations, the threshold change is stopped until it is less than the preset threshold of 0.5. Subsequently, the segmented image is subjected to multi-layer cascade filtering, including a combination of three-layer wavelet decomposition and bilateral filtering. The wavelet decomposition uses the db4 wavelet basis to obtain low-frequency approximation coefficients and high-frequency detail coefficients respectively. The bilateral filter considers the spatial distance and grayscale value difference of pixels at the same time to achieve edge-preserving denoising. In the feature extraction stage, the parallel multi-branch heterogeneous neural network contains three branches: the high-resolution branch adopts the VGG-16 structure, including 13 convolutional layers and 3 fully connected layers, with an input size of 224×224 pixels; the medium-resolution branch adopts the ResNet-50 structure, including 49 convolutional layers and 1 fully connected layer, with an input size of 112×112 pixels; the low-resolution branch adopts the MobileNetV2 structure, using depthwise separable convolution, with an input size of 56×56 pixels. The three branches realize feature interaction through cross-branch connection, and a feature exchange channel is set in the middle layer of each branch. The hierarchical adaptive attention mechanism includes two modules: channel attention and spatial attention. Channel attention generates channel descriptors through global average pooling and global maximum pooling, and spatial attention generates spatial weight maps to guide the network to focus on important areas. In the object detection and segmentation stages, the cascaded bidirectional region proposal network first generates feature maps through the forward feature extraction unit, which includes 5 feature layers of different scales, and 3 anchor boxes of different sizes are set in each layer. The reverse feature extraction unit enhances the forward features and improves the semantic information of low-level features through top-down feature transfer. During the geometric constraint optimization process, the aspect ratio of the candidate box is set to a threshold limit between 0.5 and 2.0, and the position constraint requires that the center point of the candidate box is within the valid area of ​​the image. The graph convolution network models the target area as a graph structure, where nodes represent superpixel blocks in the area, edges represent the spatial relationship between superpixel blocks, and feature propagation is performed through three layers of graph convolution.

[0028] The spatiotemporal joint attention network uses a 15-frame time window to align the features of the current frame with the historical frame through a cross spatiotemporal encoder. The multi-head collaborative attention module sets 8 attention heads, and each head independently learns different feature representations. The spatiotemporal pyramid pooling extracts features at 4 spatial scales and 3 temporal scales, and the two-stream attention fusion device processes the features of the spatial stream and the temporal stream respectively. In the multi-target tracking stage, the deep association metric first calculates the cosine similarity matrix of the target features and sets the similarity threshold of 0.75 for target association. The motion mode matching process includes four basic motion modes: the acceleration motion mode corresponds to the case where the speed change rate is greater than 20%, the uniform motion mode corresponds to the case where the speed change rate is less than 5%, the steering motion mode corresponds to the case where the direction change is greater than 30 degrees, and the stagnant motion mode corresponds to the case where the speed is close to 0. Compound motion modes include periodic motion modes (such as reciprocating motion of equipment), sudden motion modes (such as emergency stop), and interactive motion modes (such as multi-target crossing).

[0029] In the final anomaly detection stage, hierarchical contrastive learning extracts features at three levels: low-level motion features include physical quantities such as position, velocity, and acceleration; mid-level behavioral features include action sequence and trajectory features; and high-level semantic features include behavioral intention and scene semantics. Multimodal feature fusion combines temporal modality (time series data with a sampling frequency of 30 Hz), spatial modality (spatial distribution information of the target), and behavioral modality (action semantic information). The visual detection report of abnormal features contains six key parts: abnormality type (such as trajectory abnormality, speed abnormality, etc.), abnormal location (image coordinates), abnormal time (timestamp), abnormal level (level 1-5), risk assessment (low, medium, and high risk), and handling suggestions.

[0030] For example, a robot arm on an industrial production line performs a grasping task, and the input image resolution is 2560×1440 pixels. After multi-scale decomposition, images of three scales of 1280×720, 1920×1080 and 2560×1440 are obtained. The local entropy value is calculated using a 7×7 window, and the initial threshold is 138, which converges to the final threshold of 145 after 6 rounds of iterations. After multi-layer cascade filtering, the signal-to-noise ratio is improved from the original 18.3dB to 26.7dB. In the feature extraction stage, the VGG branch outputs 2048-dimensional features, the ResNet branch outputs 1024-dimensional features, and the MobileNet branch outputs 512-dimensional features. Target detection generates 312 initial candidate regions, and 94 valid candidate regions are retained after geometric constraint optimization. Spatiotemporal feature extraction uses a 15-frame time sequence window to identify a total of 3 basic motion modes (grasping, moving, and placing) and 2 compound motion modes (cyclic operation and obstacle avoidance adjustment) of the robot arm. The system detected abnormal behavior: at timestamp T=1235.6s, the robot arm experienced a sudden speed change at coordinates (1280,720), with an abnormality level of 4. The system generated a detection report containing detailed analysis and processing suggestions.

[0031] In the embodiments of the present application, by adopting a combined processing method of multi-scale decomposition and adaptive threshold segmentation, combined with multi-layer cascade filtering and nonlinear weighted fusion, the robustness and adaptability of image preprocessing can be effectively improved, ensuring that high-quality preprocessed images are obtained in complex industrial environments. The design of parallel multi-branch heterogeneous neural networks fully utilizes the advantages of different network structures, and cooperates with the hierarchical adaptive attention mechanism and cross-scale feature recalibration to achieve complementary extraction and effective fusion of multi-dimensional features, significantly enhancing the expressive power of features. The introduction of the cascaded bidirectional region proposal network improves the accuracy of target detection through candidate region generation and geometric constraint optimization. The combination of graph convolutional network and boundary dynamic adjustment mechanism further optimizes the accuracy of instance segmentation, and the spatiotemporal joint The application of attention networks realizes the effective modeling of temporal information and enhances the spatiotemporal consistency of feature extraction. The multi-target tracking strategy through deep correlation measurement and motion pattern matching not only improves the accuracy of target tracking, but also effectively handles the multi-target interaction problem in complex scenarios. Finally, the anomaly detection framework of hierarchical contrast learning and multimodal feature fusion realizes a full range of anomaly analysis from low-level physical features to high-level semantic features, and provides intuitive and explainable detection results through structured visual inspection reports. The overall solution constructs a complete intelligent machine vision inspection process through the organic combination of multiple innovative technical features. While improving the detection accuracy, it also ensures the real-time and reliability of the system, which is particularly suitable for the actual application needs of industrial sites.

[0032] In a specific embodiment, the process of executing step S101 may specifically include the following steps:

[0033] (1) Perform pyramid decomposition on the original image at scales of 0.5, 0.75, and 1 to obtain three sub-images of different scales, and perform local entropy calculation on the sub-images of different scales to obtain an initial threshold map;

[0034] (2) The initial threshold map is dynamically updated by an iterative minimum cross entropy algorithm to obtain an adaptive threshold map, and the adaptive threshold map is subjected to regional segmentation processing to obtain a regional segmentation image;

[0035] (3) performing three-layer wavelet decomposition processing on the region segmentation image to obtain low-frequency and high-frequency component images, and performing noise suppression processing on the low-frequency and high-frequency component images through bilateral filtering to obtain a denoised image;

[0036] (4) performing edge detection on the denoised image using the Sobel operator to obtain an edge feature map, and performing edge connection processing on the edge feature map using morphological operations to obtain an edge enhanced image;

[0037] (5) performing feature fusion processing on the edge enhanced image and the noise reduction image to obtain a feature fusion image, and performing weight distribution processing on the feature fusion image through a Gaussian weight function to obtain a weight coefficient map;

[0038] (6) Performing weighted summation processing on the weight coefficient map and the feature fusion image to obtain a weighted fusion image, and performing grayscale distribution adjustment processing on the weighted fusion image through histogram equalization to obtain a preprocessed image.

[0039] Specifically, the preprocessing stage of the intelligent machine vision detection method based on image processing first uses the pyramid decomposition technology to perform multi-scale processing on the original image. Pyramid decomposition is a multi-resolution analysis method that generates image sequences of different resolution levels by continuously downsampling the original image. The original image is used as the bottom layer of the pyramid (1 times), and 0.75 times and 0.5 times sub-images are generated upward in sequence. In the specific operation, the original image is smoothed by a Gaussian blur kernel, and then downsampled. The downsampling process is achieved by sampling at intervals, that is, pixels are selected at regular intervals in the horizontal and vertical directions. For the original image with a resolution of 1920×1080, two smaller sub-images of 1440×810 and 960×540 are obtained after pyramid decomposition. The calculation of local entropy value is an important indicator for evaluating the amount of information in the local area of ​​the image. For each sub-image, a sliding window of size 7×7 is set, and the probability distribution of the grayscale value of the pixel in the window is calculated, thereby obtaining the entropy value of the area. The sliding window starts from the upper left corner of the image and moves across the entire image with a step size of 1 pixel. An entropy value is calculated for each position, and finally an entropy value map of the same size as the original image is obtained. The entropy value calculation takes into account the grayscale distribution of the pixels. The larger the entropy value, the more dispersed the grayscale distribution of the area is, and the greater the amount of information.

[0040] The iterative minimum cross entropy algorithm is an adaptive threshold selection method that minimizes the cross entropy of the foreground and background categories of the image by continuously adjusting the threshold. The algorithm first selects the initial threshold based on the grayscale histogram of the image, calculates the average grayscale value of the foreground and background areas under the current threshold, and updates the threshold to the average value of the two. This process is repeated until the threshold change is less than the preset value (such as 0.5) or the maximum number of iterations is reached. After obtaining the final threshold, the pixels in the image with grayscale values ​​greater than the threshold are marked as foreground, and the pixels less than the threshold are marked as background, completing the region segmentation. The three-layer wavelet decomposition uses the Haar wavelet basis to decompose the image into a low-frequency approximate component (LL) and high-frequency detail components in three directions (LH, HL, HH). The low-frequency component contains the main energy and contour information of the image, while the high-frequency component contains edge and texture details. Wavelet decomposition divides the image into four sub-bands at each layer, and 10 sub-bands are obtained after three-layer decomposition. The bilateral filter considers both the spatial distance weight and the gray value similarity weight of pixels when performing noise suppression. The spatial weight decreases as the distance increases, and the gray value similarity weight decreases as the gray value difference increases, thereby removing noise while maintaining edge features.

[0041] The Sobel operator is a first-order differential operator that contains convolution kernels in both horizontal and vertical directions, which detect the horizontal and vertical gradients of the image respectively. When edge detection is performed on an image, the gradient values ​​in both directions are calculated respectively, and then the gradient amplitude is obtained as the edge strength. Morphological operations include two basic operations, dilation and erosion. The image is processed by designing structural elements. The dilation operation is used to expand the edge and fill the edge gap, and the erosion operation is used to refine the edge and remove noise points. Feature fusion fuses the edge enhanced image and the denoised image in a weighted average manner, and the weight is determined by the Gaussian weight function. The Gaussian weight function calculates the weight coefficient based on the spatial position and grayscale value of the pixel, and has the characteristic of smooth transition. In weighted fusion, the pixel values ​​at the corresponding positions are multiplied by the corresponding weight coefficients and then summed to obtain the fused image. Finally, the grayscale distribution of the image is adjusted by histogram equalization to improve the contrast and detail performance of the image.

[0042] For example, in the scene of industrial product surface defect detection, the input resolution is 2048×1536, and two smaller sub-images of 1536×1152 and 1024×768 are obtained after pyramid decomposition. The local entropy value is calculated using a 7×7 window, and the entropy value distribution range is between 0 and 5.7. The initial threshold of the iterative minimum cross entropy algorithm is set to 127, and after 8 iterations, it converges to the final threshold of 143, and the target area segmented accounts for 23.5% of the entire image. After three-layer wavelet decomposition, the low-frequency component energy accounts for 82.3%, and the high-frequency component energy accounts for 7.2%, 6.8% and 3.7% in three directions respectively. The spatial domain standard deviation of bilateral filtering is set to 3, and the range standard deviation is set to 25. The edge strength value obtained by Sobel operator detection is distributed between 0 and 255, and a 3×3 cross-shaped structural element is used after morphological processing. The weight of the edge image in the fusion stage is 0.3, the weight of the denoised image is 0.7, and the final histogram equalization remaps the image grayscale value to the full dynamic range of 0 to 255.

[0043] In a specific embodiment, the process of executing step S102 may specifically include the following steps:

[0044] (1) A high-resolution branch containing a VGG structure, a medium-resolution branch containing a ResNet structure, and a low-resolution branch containing a MobileNet structure are constructed for the preprocessed image to perform feature extraction processing to obtain three sets of original feature maps, and the three sets of original feature maps are subjected to feature interaction processing through cross-branch connection to obtain complementary feature maps;

[0045] (2) The complementary feature map is processed by channel attention unit to calculate channel weights to obtain channel weighted features, and the channel weighted features are processed by spatial attention unit to allocate spatial weights to obtain multi-dimensional weighted features;

[0046] (3) Performing cross-layer feature concatenation processing on the multi-dimensional weighted features through residual connection to obtain a multi-scale feature map, and performing feature resampling processing on the multi-scale feature map through pyramid pooling to obtain resampled features;

[0047] (4) The resampled features are evaluated for feature importance by an adaptive feature selector to obtain a feature importance map, and the feature importance map is subjected to feature screening by a soft gating mechanism to obtain screening features;

[0048] (5) The filtered features are transformed into reduced dimensions by a multi-layer perceptron to obtain reduced-dimensional features, and the reduced-dimensional features are normalized by feature standardization to obtain fused features.

[0049] Specifically, in the parallel multi-branch heterogeneous neural network feature extraction stage, three deep neural network branches with different structures are used to process the preprocessed image. The high-resolution branch adopts the VGG structure, which contains 5 convolution blocks, each of which contains 2-3 3×3 convolution layers and a maximum pooling layer. The input image size is 224×224 pixels, and high-level semantic features are extracted through layer-by-layer convolution and pooling operations. The medium-resolution branch adopts the ResNet structure, introduces the residual connection mechanism to avoid the gradient vanishing problem of the deep network, contains 4 residual blocks, each residual block contains 3 convolution layers, and the input size is 112×112 pixels. The low-resolution branch adopts the MobileNet structure, uses depthwise separable convolution to reduce the amount of calculation, and the input size is 56×56 pixels. The cross-branch connection realizes feature reuse through the channel interaction of the feature map, and establishes a connection channel between the intermediate layers of each branch. Specifically, after adjusting the number of channels through 1×1 convolution, the feature map output by the third convolution block of the high-resolution branch is added to the feature map output by the second residual block of the medium-resolution branch. The medium-resolution branch and the low-resolution branch are connected in the same way to form feature complementarity.

[0050] The channel attention unit first generates two channel descriptors through global average pooling and global maximum pooling respectively, and sends these two descriptors to the shared multi-layer perceptron to obtain channel weights. The spatial attention unit guides the network to focus on important areas in the image by generating an attention map of the spatial dimension. The results of channel attention and spatial attention are multiplied to obtain the final attention weight. Residual connection is used for cross-layer feature splicing, which directly adds the shallow low-level features to the deep high-level features to enrich the feature expression. Pyramid pooling performs multi-scale pooling operations on feature maps, including four scales of 1×1, 2×2, 3×3 and 6×6. The pooling results of each scale are spliced ​​after adjusting the number of channels to achieve multi-scale feature fusion.

[0051] The adaptive feature selector assigns weights to each channel by calculating the importance score of the feature channel. In specific implementation, the statistics of the spatial dimension, including mean, variance, etc., are calculated for the feature map of each channel, and the importance score is obtained through full connection layer mapping. The soft gating mechanism screens the feature channels according to the importance score. Channels with scores higher than the threshold are kept original, and channels with scores lower than the threshold are attenuated. The multilayer perceptron is used for feature dimensionality reduction. It contains two hidden layers. The first layer uses the ReLU activation function and the second layer uses the linear activation function to map high-dimensional features to the specified dimension. Feature normalization uses batch normalization to adjust the feature distribution to a standard distribution with a mean of 0 and a variance of 1.

[0052] For example, in industrial product quality inspection, the preprocessed input image is processed by three branches. The high-resolution branch VGG network inputs a 224×224 pixel image, and after 5 convolution blocks, a 14×14×512 feature map is obtained; the medium-resolution branch ResNet network inputs a 112×112 pixel image, and a 7×7×1024 feature map is obtained; the low-resolution branch MobileNet inputs a 56×56 pixel image, and a 4×4×256 feature map is obtained. After cross-branch connection, the feature dimensions are adjusted to 512, 768, and 384 channels, respectively. The weights calculated by channel attention range from 0.2 to 1.8, and spatial attention generates a 28×28 attention map. Pyramid pooling performs feature resampling at four scales to obtain feature maps of 1×1×2048, 2×2×1024, 3×3×512, and 6×6×256, which are concatenated to obtain a 3840-dimensional feature vector. The feature selection threshold is set to 0.5, and about 70% of the feature channels are retained after screening. Finally, the feature dimension is reduced to 1024 through a two-layer perceptron, and then standardized to obtain the final fusion feature.

[0053] In a specific embodiment, the process of executing step S103 may specifically include the following steps:

[0054] (1) The fused features are subjected to feature mapping processing by a forward feature extraction unit to obtain a forward feature map, and the forward feature map is subjected to feature enhancement processing by a reverse feature extraction unit to obtain a cascaded bidirectional feature;

[0055] (2) The cascaded bidirectional features are processed by the region proposal generator to generate the initial region to obtain the initial candidate region, and the initial candidate region is processed by the geometric feature extractor to extract the shape feature to obtain the geometric feature;

[0056] (3) Geometric constraints are processed on the geometric features through the constraint rule library to obtain a constraint candidate box, and the constraint candidate box is optimized by the region optimizer to obtain a target candidate region;

[0057] (4) Performing node feature extraction processing on the target candidate area through the graph node builder to obtain graph node features, and performing connection relationship construction processing on the graph node features through the graph edge generator to obtain graph structure features;

[0058] (5) Performing feature propagation processing on the graph structure features through the graph convolution layer to obtain graph convolution features, and performing boundary position prediction processing on the graph convolution features through the boundary dynamic predictor to obtain a boundary prediction map;

[0059] (6) The boundary prediction map is fine-tuned by a dynamic adjustment unit to obtain an adjusted boundary, and the adjusted boundary is divided into regions by an instance segmentor to obtain a segmentation result.

[0060] Specifically, in the cascaded bidirectional region proposal network, the forward feature extraction unit first performs feature mapping on the fused features and processes them using 5 feature layers of different scales. Each feature layer contains a combination of 3×3 convolutional layers and 1×1 convolutional layers, which capture local features and global context information respectively. The reverse feature extraction unit enhances feature expression through a top-down feature transfer path, and transfers high-level semantic information to low-level feature maps layer by layer to achieve feature enhancement. The region proposal generator generates candidate regions based on multi-scale feature maps, sets anchor boxes of 3 different sizes and 3 different aspect ratios on each feature layer, and divides positive and negative samples according to the preset IoU threshold (set to 0.7). The geometric feature extractor calculates shape-related features of the initial candidate region, including geometric properties such as area, perimeter, aspect ratio, and compactness, to form a feature vector describing the shape characteristics of the candidate region.

[0061] The constraint rule library defines a series of geometric constraints, including the area range of the candidate box (0.01% to 90% of the image area), the aspect ratio range (between 0.5 and 2.0), the boundary position constraint (the center point of the box must be within the valid area of ​​the image), etc. The region optimizer adjusts the candidate boxes according to these constraints, removes the candidate boxes that do not meet the constraints, and fine-tunes the position and size of the remaining candidate boxes. The graph node builder divides the target candidate area into superpixel blocks, each superpixel block as a node of the graph, and extracts the node's color, texture, and position as node features. The graph edge generator constructs a connection relationship based on the spatial adjacency relationship and feature similarity between nodes. Spatial adjacency uses 8-neighborhood connection, feature similarity is measured by cosine distance, and the similarity threshold is set to 0.8.

[0062] The boundary position prediction uses the following formula:

[0063] P(x,y)=σ(F gcn (x,y)·W b +b b )

[0064] Among them, P(x,y) represents the probability of the boundary at the position (x,y), and F gcn is the graph convolution feature, W b and b b is the prediction layer parameter, and σ is the sigmoid activation function.

[0065] B(x,y)=α·P(x,y)+(1-α)·E(x,y)

[0066] Among them, B(x,y) is the final boundary position, E(x,y) is the edge feature map, and α is the balance factor.

[0067] The dynamic adjustment unit refines the boundary position according to the boundary probability distribution of the local area and gradually adjusts the boundary position in an iterative manner. The instance segmenter uses the adjusted boundary as a constraint and uses the region growing algorithm to segment the target area to obtain the final segmentation result.

[0068] For example, in the industrial parts detection scenario, the input feature map size is 56×56×256, and the forward feature extraction obtains 5 scale feature maps, with sizes of 56×56×256, 28×28×512, 14×14×512, 7×7×256, and 4×4×256. After reverse feature enhancement, the feature dimension remains unchanged but the feature expression is richer. The region proposal generator generates a total of 15912 initial candidate boxes on 5 feature layers, and retains 2048 candidate boxes after non-maximum suppression (IoU threshold 0.7). The geometric feature extraction calculates the geometric features of these candidate boxes: the area ranges from 100 to 10000 pixels, the aspect ratio is distributed between 0.5 and 2.0, and the compactness (perimeter squared / area) is between 12 and 20. After constraint rule screening, 643 valid candidate boxes are retained. The graph structure construction divides each candidate region into an average of 50 superpixel nodes, extracts a 128-dimensional feature vector from each node, and the average number of connections between nodes is 180. The output feature dimensions of the three-layer graph convolutional network are 128, 256, and 512, respectively. The size of the probability map obtained by boundary prediction is the same as the input feature map. After three rounds of iterative optimization, the boundary positioning accuracy reaches the pixel level. The final instance segmentation is completed in each candidate region to obtain the precise boundary of the target.

[0069] In a specific embodiment, the process of executing step S104 may specifically include the following steps:

[0070] (1) The segmentation results are processed into time series frames by the time dimension slicer to obtain a time series feature sequence, and the time series feature sequence is processed by the dynamic self-attention unit to calculate the inter-frame correlation to obtain a time series correlation matrix;

[0071] (2) The temporal correlation matrix is ​​encoded with the cross-temporal encoder to obtain the initial spatiotemporal features, and the initial spatiotemporal features are enhanced by the multi-head collaborative attention module to obtain enhanced spatiotemporal features;

[0072] (3) The enhanced spatiotemporal features are subjected to multi-scale feature extraction processing through the spatiotemporal pyramid pooling unit to obtain multi-scale spatiotemporal features, and the multi-scale spatiotemporal features are subjected to feature integration processing through the dual-stream attention fusion unit to obtain fused spatiotemporal features;

[0073] (4) The fused spatiotemporal features are processed by temporal dependency modeling through a hierarchical spatiotemporal memory network to obtain spatiotemporal memory features, and the spatiotemporal memory features are processed by scene association through a context perception module to obtain scene context features;

[0074] (5) The scene context features are aggregated by a densely connected aggregator to obtain aggregated features, and the aggregated features are optimized by a spatiotemporal feature recalibration unit to obtain spatiotemporal context features.

[0075] Specifically, in the intelligent machine vision detection method based on image processing, the temporal feature extraction first processes the segmentation results through the time dimension slicer, sets the time window length to 15 frames, and the sliding step length to 1 frame, and divides the continuous image sequence into multiple temporal segments. The image features in each temporal segment are linearly projected to obtain a feature vector with the same dimension, forming a temporal feature sequence. The dynamic self-attention unit adopts the following calculation method:

[0076]

[0077] Among them, Q, K, and V represent the query matrix, key matrix, and value matrix respectively, and d k is the feature dimension, and the softmax function is used to normalize the attention weight. The correlation strength matrix between frames is obtained through this calculation. The cross-spatiotemporal encoder performs spatiotemporal joint modeling of the features in the temporal correlation matrix, and includes a multi-layer encoder structure. Each encoder layer contains a self-attention layer and a feedforward network layer to achieve interactive fusion of temporal information and spatial information. The multi-head collaborative attention module sets 8 attention heads, each head independently learns different feature representations, and then splices the outputs of multiple heads and undergoes linear transformation to obtain enhanced spatiotemporal features. The spatiotemporal pyramid pooling unit performs multi-scale feature extraction in both time and space dimensions. Pooling windows of 1, 3, and 5 frames are used in the time dimension, and pooling kernels of 1×1, 2×2, and 4×4 are used in the spatial dimension to generate feature maps of different scales. The two-stream attention fusion unit performs attention calculations on the temporal and spatial stream features respectively, and then fuses the features of the two streams by weighted summation.

[0078] The hierarchical spatiotemporal memory network adopts a three-layer LSTM structure. The hidden layer dimension of each LSTM unit is 512. The gating mechanism is used to selectively memorize and forget the temporal information. The context-aware module calculates the similarity between the current moment features and the historical features, aggregates the features according to the similarity weight, and realizes scene-level semantic understanding. The dense connection aggregator adopts a dense connection structure similar to DenseNet to directly connect features at different levels to ensure full utilization of information. The spatiotemporal feature recalibration unit evaluates the importance of features at the channel level and the spatial level, and dynamically adjusts the feature weights.

[0079] For example, in the product quality inspection scenario on an industrial production line, the input video frame rate is 30fps, and 15 frames of image sequences are processed each time. The feature dimension of the segmentation result of each frame is 256×256×64, and after slicing in the time dimension, a 15×256×256×64 four-dimensional tensor is obtained. The dynamic self-attention calculation obtains a 15×15 inter-frame correlation matrix, which reflects the degree of correlation between different time frames. The 4-layer structure of the cross-spatiotemporal encoder contains 8 attention heads per layer, and the feature dimension is 512. Spatiotemporal pyramid pooling extracts features at 3 time scales and 3 spatial scales, generating a total of 9 feature representations at different scales. The weight ratio of the time stream and the spatial stream in the two-stream attention fusion is 0.4:0.6. The hidden layer state dimensions of the three-layer LSTM are 512, 256, and 128, respectively, and the 15-frame sequence is modeled to obtain the temporal dependency feature. The final output spatiotemporal context feature dimension is 256, which contains the temporal change law and spatial context information of the target. Through this multi-level spatiotemporal feature extraction and fusion, the dynamic change characteristics of the product during the production process can be accurately captured, providing reliable feature representation for subsequent anomaly detection.

[0080] In a specific embodiment, the process of executing step S105 may specifically include the following steps:

[0081] (1) Performing feature mapping processing on the spatiotemporal context features through a deep embedding network to obtain a deep embedding vector, and performing association measurement processing on the deep embedding vector through a cosine similarity calculator to obtain a target association matrix;

[0082] (2) performing target pairing processing on the target association matrix through a multi-target matcher to obtain target tracking pairs, and performing motion state extraction processing on the target tracking pairs through a time series state estimator to obtain a motion state sequence;

[0083] (3) extracting basic motion features from the motion state sequence through a motion pattern decomposer to obtain basic motion features including acceleration motion pattern, uniform motion pattern, turning motion pattern and stagnant motion pattern, and constructing composite motion features from the basic motion features through a pattern combiner to obtain composite motion features including periodic motion pattern, sudden motion pattern and interactive motion pattern;

[0084] (4) performing motion pattern matching processing on the composite motion features through a complementary pattern matcher to obtain a matching pattern sequence, and performing trend prediction processing on the matching pattern sequence through a motion trend analyzer to obtain a motion trend feature;

[0085] (5) The motion trend features are processed by the trajectory generator to construct the trajectory to obtain the initial trajectory, and the initial trajectory is optimized by the trajectory smoother to obtain the trajectory prediction result.

[0086] Specifically, in multi-target tracking and motion pattern analysis, the spatiotemporal context features are first nonlinearly mapped through a deep embedding network. The deep embedding network consists of 4 fully connected layers, with the output dimensions of each layer being 1024, 512, 256, and 128, respectively. The ReLU activation function is used, and the stability of the feature distribution is maintained through batch normalization. The cosine similarity calculator calculates the similarity between two targets based on the feature vector normalized by the L2 norm, and constructs an N×N target association matrix, where N is the number of targets detected in the current frame. The multi-target matcher uses the Hungarian algorithm to optimize target association, converting the association problem into a bipartite graph maximum weight matching problem. The similarity threshold is set to 0.75, and the association between targets is established only when the similarity exceeds the threshold. The temporal state estimator uses a Kalman filter to track the motion state of the target. The state vector contains the target's position coordinates (x, y), velocity (vx, vy), and acceleration (ax, ay), and the state estimate is updated once per frame.

[0087] The motion mode decomposer extracts four basic motion modes according to the state sequence: acceleration motion mode (acceleration greater than the threshold 2mm / s 2 ), uniform motion mode (acceleration is less than the threshold value of 0.5 mm / s 2 ), turning motion mode (direction change greater than 30 degrees) and stagnant motion mode (speed less than the threshold of 1mm / s). The pattern combiner identifies three types of compound motion patterns based on the temporal combination rules of basic motion patterns: periodic motion pattern (basic pattern repeats at fixed time intervals), sudden motion pattern (basic pattern changes dramatically) and interactive motion pattern (motion patterns of multiple targets have spatiotemporal correlation).

[0088] The complementary pattern matcher uses the dynamic time warping algorithm (DTW) to calculate the matching degree between the current motion sequence and the standard pattern in the preset pattern library, and selects the pattern with the highest matching degree as the motion pattern of the current state. The motion trend analyzer predicts the motion trend of the next 5 frames based on the motion sequence of the past 15 frames through the autoregressive model, taking into account the changing laws of characteristics such as speed, acceleration, and motion direction. The trajectory generator uses cubic spline interpolation to generate a smooth trajectory curve based on the predicted motion trend. The interpolation points are selected based on the state estimation results at key moments. The trajectory smoother eliminates noise in the trajectory through sliding window averaging and Gaussian filtering. The window size is set to 5 frames and the Gaussian kernel standard deviation is 1.5.

[0089] For example: In the industrial robot collaboration scenario, the input spatiotemporal context feature dimension is 256, which is mapped to a 128-dimensional feature vector through the deep embedding network. Assuming that 5 targets are detected in the current frame, a 5×5 target association matrix is ​​constructed. The motion sequence of target 1 is obtained through state estimation: position (100,100) and speed (5,0) at t=0, position (105,100) and speed (5,0) at t=1, position (110,100) and speed (5,5) at t=2, showing the characteristics of uniform linear motion turning into accelerated steering. Basic motion feature extraction shows: t=0 to t=1 is a uniform motion mode (acceleration 0.1mm / s 2 ), t = 1 to t = 2 is the turning motion mode (direction change 45 degrees) and the acceleration motion mode (acceleration 2.2mm / s 2 ). Pattern combination analysis found that this is a typical mutation motion pattern. The DTW algorithm calculated the minimum regularized distance from the preset pattern to be 0.82, confirming the robot's obstacle avoidance operation mode. Motion trend analysis predicts the position sequence of the next 5 frames: (115,105), (120,110), (125,115), (130,120), (135,125), and the corresponding speed sequence: (5,5), (5,5), (5,5), (5,5), (5,5). The trajectory generator generates a smooth trajectory based on these predicted points. After trajectory smoothing, the deviation of each position does not exceed 2 pixels, and the final trajectory prediction result is obtained.

[0090] In a specific embodiment, the process of executing step S106 may specifically include the following steps:

[0091] (1) The trajectory prediction results are hierarchically decomposed through a multi-level feature extractor to obtain a hierarchical feature set including low-level motion features, mid-level behavior features, and high-level semantic features. The hierarchical feature set is then enhanced through a positive sample enhancer to obtain enhanced positive samples.

[0092] (2) performing feature encoding processing on the enhanced positive sample and the preset abnormal sample library through a comparative feature encoder to obtain a comparative feature vector, and performing similarity calculation processing on the comparative feature vector through a hierarchical metric calculator to obtain a hierarchical similarity matrix;

[0093] (3) The hierarchical similarity matrix is ​​subjected to temporal modality, spatial modality, and behavioral modality feature extraction processing by a multimodal feature extractor to obtain a multimodal feature group, and the multimodal feature group is subjected to feature alignment processing by a cross-modal aligner to obtain an alignment feature;

[0094] (4) The alignment features are quantified by an abnormality calculator to obtain an abnormality score, and the abnormality score is graded by a multi-threshold classifier to obtain an abnormality grade;

[0095] (5) The abnormal level is processed by the spatiotemporal locator to locate the abnormal event, and the abnormal event description is processed by the scene association analyzer to perform context analysis and obtain the scene understanding result;

[0096] (6) The scene understanding results are processed by the risk assessor to perform risk level assessment to obtain the risk assessment results, and the risk assessment results are processed by the report generator to fill in the report template to obtain a visual inspection report containing the anomaly type, anomaly location, anomaly time, anomaly level, risk assessment and processing suggestions.

[0097] Specifically, the trajectory prediction results are hierarchically decomposed by a multi-level feature extractor to extract features at three levels: low-level motion features include the target's position coordinate sequence, velocity vector sequence, and acceleration vector sequence, each of which contains data at 15 time points; mid-level behavioral features include the curvature change, velocity change rate, and direction change rate of the motion trajectory, which are calculated using a sliding window; high-level semantic features include motion pattern types, state transition sequences, and target interaction relationships, which are mapped through a semantic encoder. The positive sample enhancer uses data enhancement technology to randomly rotate (±15 degrees), translate (±10 pixels), and scale the speed (0.8-1.2 times) normal samples to expand the number of positive samples. The contrast feature encoder encodes the enhanced positive samples and samples in the abnormal sample library into feature vectors of uniform dimension. The abnormal sample library contains various pre-annotated abnormal patterns, such as trajectory mutations, velocity anomalies, acceleration anomalies, etc. The hierarchical metric calculator calculates the similarity between samples at three levels: the low level uses Euclidean distance to measure the differences in position and motion parameters, the middle level uses DTW algorithm to measure the similarity of behavior sequences, and the high level uses cosine distance to measure the similarity of semantic features.

[0098] The multimodal feature extractor further analyzes the hierarchical similarity matrix and extracts features from the temporal modality (time series data with a sampling frequency of 30Hz), the spatial modality (spatial distribution and motion range of the target), and the behavioral modality (action semantics and interaction mode). The cross-modal aligner aligns and fuses the features of different modalities through the attention mechanism to ensure the consistency of feature representation. The anomaly calculator calculates the anomaly score based on the aligned features and uses the isolation forest algorithm to evaluate the degree of anomaly of the sample. The multi-threshold classifier divides the anomaly score into 5 levels: level 1 (score <0.3, normal), level 2 (0.3-0.5, slight anomaly), level 3 (0.5-0.7, moderate anomaly), level 4 (0.7-0.9, severe anomaly), and level 5 (>0.9, dangerous anomaly).

[0099] The spatiotemporal locator determines the specific spatiotemporal location of the abnormal event according to the anomaly level, and records the timestamp and image coordinates of the anomaly. The scene association analyzer considers the contextual information of the abnormal event and analyzes the cause and scope of the anomaly. The risk assessor assesses the risk level according to the anomaly level, scope of impact and duration, and uses a rule-based scoring mechanism: impact range score (1-5 points) × anomaly level (1-5 points) × duration weight (1-2 points) to calculate the final risk score. The report generator integrates all analysis results into a detection report in a standard format. The report contains six key parts: anomaly type (such as trajectory anomaly, speed anomaly, etc.), anomaly location (pixel coordinates), anomaly time (timestamp), anomaly level (1-5 levels), risk assessment (low, medium and high risk) and handling suggestions.

[0100] For example, in an industrial robot collaboration scenario, the motion of a six-axis industrial robot is analyzed when performing a handling task. Low-level feature extraction shows that the position sequence moves from workbench A (coordinates: x=1200mm, y=800mm, z=500mm) to workbench B (coordinates: x=2000mm, y=1500mm, z=500mm), the robot terminal speed suddenly changes from the standard operating speed of 250mm / s to 750mm / s, and the instantaneous acceleration peak reaches 1800mm / s 2 , far exceeding the set safety acceleration threshold of 1000mm / s 2. The middle-level feature analysis found that during the robot's motion, the TCP (tool center point) posture angle changed dramatically in the 8th frame, deflecting more than 45 degrees from the vertical, and the speed change rate of joints 3 and 4 reached 180%, which was beyond the normal range. The high-level semantic features identified a "posture abnormality caused by load mutation" pattern. After feature encoding, it was compared with the abnormal sample library, and the calculated similarity matrix showed that the similarity with the "workpiece falling abnormality" reached 0.87. Multimodal analysis showed that: in the time mode, a speed mutation was detected at t=58.5s; in the spatial mode, the TCP trajectory deviated from the preset straight path by more than 150mm; in the behavioral mode, the "uniform and stable handling" operation specification was violated. The abnormality calculation score was 0.83, which was judged as a level 4 abnormality. The spatiotemporal positioning determined that the abnormality occurred at the timestamp 58.5s, the position coordinates (1500mm, 1200mm, 500mm), and it was in the transition area between the two workbenches. The scene association analysis found that the abnormal position was located on the cross path of the multi-robot collaborative work area and was close to the manual operation area.

[0101] Based on the risk assessment matrix calculation: impact range score 4 points (affecting adjacent workstations) × abnormality level 4 points (may cause damage to the workpiece) × duration coefficient 1.5 (abnormal duration > 2s) = total score 24 points, which is determined to be a high risk level. The generated test report contains:

[0102] Abnormal type: abnormal workpiece handling posture;

[0103] Abnormal location: TCP coordinates (1500mm, 1200mm, 500mm);

[0104] Abnormal time: 2024-02-15 10:58:30.5;

[0105] Abnormality level: Level 4 (serious abnormality);

[0106] Risk assessment: High risk (score 24 points).

[0107] Handling suggestions: Immediately stop the handling task of the work unit; check the end effector gripping mechanism; recalibrate the load parameters; recheck whether the workpiece weight meets the specifications.

[0108] The above describes the intelligent machine vision detection method based on image processing in the embodiment of the present application. The following describes the intelligent machine vision detection device based on image processing in the embodiment of the present application. Figure 2 In the embodiment of the present application, an embodiment of an intelligent machine vision detection device based on image processing includes:

[0109] The segmentation module 201 is used to perform multi-scale decomposition and adaptive threshold segmentation processing on the original image to obtain a segmented image, and perform multi-layer cascade filtering and nonlinear weighted fusion processing on the segmented image to obtain a pre-processed image;

[0110] A fusion module 202 is used to perform complementary feature extraction processing on the pre-processed image through a parallel multi-branch heterogeneous neural network to obtain multi-dimensional features, and perform feature fusion processing on the multi-dimensional features through a hierarchical adaptive attention mechanism and cross-scale feature recalibration to obtain fused features;

[0111] The optimization module 203 is used to generate candidate regions and perform geometric constraint optimization processing on the fused features through a cascaded bidirectional region proposal network to obtain a target candidate region, and perform instance segmentation processing on the target candidate region through a graph convolution network and a boundary dynamic adjustment mechanism to obtain a segmentation result;

[0112] An extraction module 204 is used to perform feature extraction processing on the segmentation result through a spatiotemporal joint attention network to obtain spatiotemporal context features;

[0113] A tracking module 205 is used to perform multi-target tracking processing on the spatiotemporal context features through deep association measurement and motion pattern matching to obtain a trajectory prediction result;

[0114] The generation module 206 is used to perform abnormal pattern modeling processing on the trajectory prediction result through hierarchical contrast learning and multimodal feature fusion to obtain abnormal features, and generate a visual inspection report according to the abnormal features.

[0115] Through the collaborative cooperation of the above components, by adopting a combined processing method of multi-scale decomposition and adaptive threshold segmentation, combined with multi-layer cascade filtering and nonlinear weighted fusion, the robustness and adaptability of image preprocessing can be effectively improved, ensuring high-quality preprocessed images in complex industrial environments. The design of parallel multi-branch heterogeneous neural networks fully utilizes the advantages of different network structures, and cooperates with the hierarchical adaptive attention mechanism and cross-scale feature recalibration to achieve complementary extraction and effective fusion of multi-dimensional features, significantly enhancing the expression ability of features. The introduction of the cascaded bidirectional region proposal network improves the accuracy of target detection through candidate region generation and geometric constraint optimization. The combination of graph convolutional network and boundary dynamic adjustment mechanism further optimizes the accuracy of instance segmentation. The application of spatiotemporal joint attention network realizes the effective modeling of temporal information and enhances the spatiotemporal consistency of feature extraction. The multi-target tracking strategy through deep correlation measurement and motion pattern matching not only improves the accuracy of target tracking, but also effectively handles the multi-target interaction problem in complex scenarios. Finally, the anomaly detection framework of hierarchical contrast learning and multimodal feature fusion realizes a full range of anomaly analysis from low-level physical features to high-level semantic features, and provides intuitive and explainable detection results through structured visual inspection reports. The overall solution constructs a complete intelligent machine vision inspection process through the organic combination of multiple innovative technical features. While improving the detection accuracy, it also ensures the real-time and reliability of the system, which is particularly suitable for the actual application needs of industrial sites.

[0116] The present application also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the image processing-based intelligent machine vision detection method.

[0117] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0118] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0119] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An intelligent machine vision detection method based on image processing, characterized in that: The intelligent machine vision detection method based on image processing includes: Performing multi-scale decomposition and adaptive threshold segmentation processing on the original image to obtain a segmented image, and performing multi-layer cascade filtering and nonlinear weighted fusion processing on the segmented image to obtain a preprocessed image; Performing complementary feature extraction processing on the preprocessed image through a parallel multi-branch heterogeneous neural network to obtain multidimensional features, and performing feature fusion processing on the multidimensional features through a hierarchical adaptive attention mechanism and cross-scale feature recalibration to obtain fused features; The fused features are subjected to candidate region generation and geometric constraint optimization processing through a cascaded bidirectional region proposal network to obtain a target candidate region, and the target candidate region is subjected to instance segmentation processing through a graph convolutional network and a boundary dynamic adjustment mechanism to obtain a segmentation result; Performing feature extraction processing on the segmentation result through a spatiotemporal joint attention network to obtain spatiotemporal context features; Performing multi-target tracking processing on the spatiotemporal context features through deep correlation measurement and motion pattern matching to obtain trajectory prediction results; The trajectory prediction result is processed by abnormal pattern modeling through hierarchical contrast learning and multimodal feature fusion to obtain abnormal features, and a visual inspection report is generated according to the abnormal features.

2. The intelligent machine vision detection method based on image processing according to claim 1 is characterized in that: The method of performing multi-scale decomposition and adaptive threshold segmentation processing on the original image to obtain a segmented image, and performing multi-layer cascade filtering and nonlinear weighted fusion processing on the segmented image to obtain a pre-processed image includes: Performing pyramid decomposition processing on the original image at scale ratios of 0.5, 0.75 and 1 to obtain three sub-images of different scales, and performing local entropy value calculation processing on the three sub-images of different scales to obtain an initial threshold map; Performing a threshold dynamic update process on the initial threshold map by an iterative minimum cross entropy algorithm to obtain an adaptive threshold map, and performing a region segmentation process on the adaptive threshold map to obtain a region segmentation image; Performing three-layer wavelet decomposition processing on the region segmented image to obtain low-frequency and high-frequency component images, and performing noise suppression processing on the low-frequency and high-frequency component images through bilateral filtering to obtain a denoised image; Performing edge detection processing on the denoised image by using a Sobel operator to obtain an edge feature map, and performing edge connection processing on the edge feature map by using a morphological operation to obtain an edge enhanced image; Performing feature fusion processing on the edge enhanced image and the noise reduction image to obtain a feature fusion image, and performing weight distribution processing on the feature fusion image through a Gaussian weight function to obtain a weight coefficient map; The weight coefficient map and the feature fusion image are subjected to weighted summation processing to obtain a weighted fusion image, and the weighted fusion image is subjected to grayscale distribution adjustment processing through histogram equalization to obtain the preprocessed image.

3. The intelligent machine vision detection method based on image processing according to claim 1 is characterized in that: The pre-processed image is subjected to complementary feature extraction processing through a parallel multi-branch heterogeneous neural network to obtain multi-dimensional features, and the multi-dimensional features are subjected to feature fusion processing through a hierarchical adaptive attention mechanism and cross-scale feature recalibration to obtain fused features, including: Constructing a high-resolution branch including a VGG structure, a medium-resolution branch including a ResNet structure, and a low-resolution branch including a MobileNet structure for the preprocessed image, respectively, to perform feature extraction processing to obtain three groups of original feature maps, and performing feature interaction processing on the three groups of original feature maps through cross-branch connection to obtain complementary feature maps; The complementary feature map is processed by a channel attention unit to calculate channel weights to obtain channel weighted features, and the channel weighted features are processed by a spatial attention unit to allocate spatial weights to obtain multidimensional weighted features; Performing cross-layer feature concatenation processing on the multi-dimensional weighted features through residual connection to obtain a multi-scale feature map, and performing feature resampling processing on the multi-scale feature map through pyramid pooling to obtain resampled features; Performing feature importance evaluation processing on the resampled features through an adaptive feature selector to obtain a feature importance map, and performing feature screening processing on the feature importance map through a soft gating mechanism to obtain screening features; The screening features are subjected to dimensionality reduction transformation processing through a multi-layer perceptron to obtain dimensionality reduction features, and the dimensionality reduction features are normalized through feature standardization to obtain the fusion features.

4. The intelligent machine vision detection method based on image processing according to claim 1 is characterized in that: The fusion features are subjected to candidate region generation and geometric constraint optimization processing through a cascaded bidirectional region proposal network to obtain a target candidate region, and the target candidate region is subjected to instance segmentation processing through a graph convolution network and a boundary dynamic adjustment mechanism to obtain a segmentation result, including: Performing feature mapping processing on the fused features through a forward feature extraction unit to obtain a forward feature map, and performing feature enhancement processing on the forward feature map through a reverse feature extraction unit to obtain a cascaded bidirectional feature; Performing initial region generation processing on the cascaded bidirectional features through a region proposal generator to obtain an initial candidate region, and performing shape feature extraction processing on the initial candidate region through a geometric feature extractor to obtain a geometric feature; Performing geometric constraint processing on the geometric features through a constraint rule library to obtain a constraint candidate box, and performing boundary optimization processing on the constraint candidate box through a region optimizer to obtain the target candidate region; Performing node feature extraction processing on the target candidate area through a graph node builder to obtain graph node features, and performing connection relationship construction processing on the graph node features through a graph edge generator to obtain graph structure features; Performing feature propagation processing on the graph structure features through a graph convolution layer to obtain graph convolution features, and performing boundary position prediction processing on the graph convolution features through a boundary dynamic predictor to obtain a boundary prediction graph; The boundary prediction map is subjected to boundary fine-tuning processing by a dynamic adjustment unit to obtain an adjusted boundary, and the adjusted boundary is subjected to region segmentation processing by an instance segmentor to obtain the segmentation result.

5. The intelligent machine vision detection method based on image processing according to claim 1 is characterized in that: The segmentation result is subjected to feature extraction processing through a spatiotemporal joint attention network to obtain spatiotemporal context features, including: The segmentation result is processed by time dimension slicer to obtain a time series feature sequence, and the time series feature sequence is processed by dynamic self-attention unit to obtain a time series correlation matrix; Performing spatiotemporal feature encoding processing on the temporal association matrix through a cross spatiotemporal encoder to obtain initial spatiotemporal features, and performing feature enhancement processing on the initial spatiotemporal features through a multi-head collaborative attention module to obtain enhanced spatiotemporal features; Performing multi-scale feature extraction processing on the enhanced spatiotemporal features through a spatiotemporal pyramid pooling unit to obtain multi-scale spatiotemporal features, and performing feature integration processing on the multi-scale spatiotemporal features through a dual-stream attention fuser to obtain fused spatiotemporal features; Performing temporal dependency modeling processing on the fused spatiotemporal features through a hierarchical spatiotemporal memory network to obtain spatiotemporal memory features, and performing scene association processing on the spatiotemporal memory features through a context perception module to obtain scene context features; The scene context features are subjected to feature aggregation processing through a densely connected aggregator to obtain aggregated features, and the aggregated features are subjected to feature optimization processing through a spatiotemporal feature recalibration unit to obtain the spatiotemporal context features.

6. The intelligent machine vision detection method based on image processing according to claim 1 is characterized in that: The multi-target tracking process is performed on the spatiotemporal context features through deep association measurement and motion pattern matching to obtain trajectory prediction results, including: Performing feature mapping processing on the spatiotemporal context features through a deep embedding network to obtain a deep embedding vector, and performing association measurement processing on the deep embedding vector through a cosine similarity calculator to obtain a target association matrix; Performing target pairing processing on the target association matrix through a multi-target matcher to obtain a target tracking pair, and performing motion state extraction processing on the target tracking pair through a time series state estimator to obtain a motion state sequence; The motion state sequence is subjected to basic motion feature extraction processing by a motion pattern decomposer to obtain basic motion features including an acceleration motion pattern, a uniform motion pattern, a turning motion pattern and a stagnant motion pattern, and the basic motion features are subjected to composite motion feature construction processing by a pattern combiner to obtain composite motion features including a periodic motion pattern, a sudden motion pattern and an interactive motion pattern; Performing motion pattern matching processing on the composite motion feature through a complementary pattern matcher to obtain a matching pattern sequence, and performing trend prediction processing on the matching pattern sequence through a motion trend analyzer to obtain a motion trend feature; The motion trend feature is subjected to trajectory construction processing by a trajectory generator to obtain an initial trajectory, and the initial trajectory is subjected to trajectory optimization processing by a trajectory smoother to obtain the trajectory prediction result.

7. The intelligent machine vision detection method based on image processing according to claim 1 is characterized in that: The abnormal pattern modeling processing is performed on the trajectory prediction result through hierarchical contrast learning and multimodal feature fusion to obtain abnormal features, and a visual inspection report is generated according to the abnormal features, including: The trajectory prediction result is subjected to hierarchical decomposition processing by a multi-level feature extractor to obtain a hierarchical feature set including low-level motion features, middle-level behavior features and high-level semantic features, and the hierarchical feature set is subjected to feature enhancement processing by a positive sample enhancer to obtain an enhanced positive sample; Performing feature encoding processing on the enhanced positive sample and the preset abnormal sample library through a comparison feature encoder to obtain a comparison feature vector, and performing similarity calculation processing on the comparison feature vector through a hierarchical metric calculator to obtain a hierarchical similarity matrix; Performing temporal modality, spatial modality and behavioral modality feature extraction processing on the hierarchical similarity matrix through a multimodal feature extractor to obtain a multimodal feature group, and performing feature alignment processing on the multimodal feature group through a cross-modal aligner to obtain alignment features; The alignment features are subjected to an abnormality degree quantification processing by an abnormality calculator to obtain an abnormality degree score, and the abnormality degree score is subjected to a grade classification processing by a multi-threshold classifier to obtain an abnormality grade; Performing abnormal event location processing on the abnormal level through a spatiotemporal locator to obtain an abnormal event description, and performing context analysis processing on the abnormal event description through a scene association analyzer to obtain a scene understanding result; The scene understanding result is processed by a risk assessor to perform risk level assessment to obtain a risk assessment result, and the risk assessment result is processed by a report generator to fill in a report template to obtain the visual inspection report including the abnormality type, abnormality location, abnormality time, abnormality level, risk assessment and processing suggestions.

8. An intelligent machine vision detection device based on image processing, used to implement the intelligent machine vision detection method based on image processing as described in any one of claims 1 to 7, characterized in that: The intelligent machine vision detection device based on image processing comprises: The segmentation module is used to perform multi-scale decomposition and adaptive threshold segmentation processing on the original image to obtain a segmented image, and perform multi-layer cascade filtering and nonlinear weighted fusion processing on the segmented image to obtain a pre-processed image; A fusion module is used to perform complementary feature extraction processing on the pre-processed image through a parallel multi-branch heterogeneous neural network to obtain multi-dimensional features, and perform feature fusion processing on the multi-dimensional features through a hierarchical adaptive attention mechanism and cross-scale feature recalibration to obtain fused features; An optimization module, used to generate candidate regions and perform geometric constraint optimization processing on the fused features through a cascaded bidirectional region proposal network to obtain a target candidate region, and perform instance segmentation processing on the target candidate region through a graph convolution network and a boundary dynamic adjustment mechanism to obtain a segmentation result; An extraction module, used for performing feature extraction processing on the segmentation result through a spatiotemporal joint attention network to obtain spatiotemporal context features; A tracking module, used to perform multi-target tracking processing on the spatiotemporal context features through deep association measurement and motion pattern matching to obtain trajectory prediction results; The generation module is used to perform abnormal pattern modeling processing on the trajectory prediction result through hierarchical contrast learning and multimodal feature fusion to obtain abnormal features, and generate a visual detection report based on the abnormal features.

9. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the intelligent machine vision detection method based on image processing as described in any one of claims 1 to 7 is implemented.

Images