Internet of Things-based AI-powered image detection methods, systems, and storage media

Abstract

This invention relates to the field of visual monitoring technology for IoT devices, and discloses an AI-based image detection method, system, and storage medium for IoT. The method synchronously receives image and environmental parameter data streams, performs temporal slicing on the image stream, and extracts deep visual features from the current slice. The environmental data stream is decomposed into waveforms, and corresponding environmental state segments are extracted based on image timestamps. A dynamic difference spectrum is generated by calculating the differences between current and past image features, and visual features are coupled and encoded with environmental segments to obtain enhanced visual representations. Using both as inputs, a cascaded diagnostic inference engine is driven to perform anomaly localization, pattern matching, and defect inference, ultimately outputting detection conclusions. This method can capture gradual changes in device status and achieve more accurate fault diagnosis under complex environmental fluctuations.

Description

Technical Field

[0001] This invention relates to the field of visual monitoring technology for Internet of Things (IoT) devices, specifically to an AI-based image detection method, system, and storage medium based on the Internet of Things. Background Technology

[0002] In the field of intelligent monitoring of industrial equipment, the comprehensive use of IoT images and environmental sensors for status recognition is a common solution. Existing technologies typically perform feature analysis and defect classification on single-frame real-time images or image sets over a fixed time period. For synchronously acquired environmental parameters such as temperature and vibration, the conventional approach is to treat them as independent features and directly stitch them together or simply fuse them with image features to aid in judgment.

[0003] These existing solutions have shortcomings. Analysis based on static or short-term images is difficult to effectively distinguish between long-term equipment performance degradation, intermittent anomalies, and image changes caused by instantaneous external interference, leading to false alarms. Environmental data itself contains trends, periods, and noise. Combining it as a whole with image features in a coarse-grained manner cannot accurately establish a dynamic correlation model between environmental conditions and equipment visual performance at a specific point in time, reducing the reliability of diagnosis under complex and variable operating conditions.

[0004] A method is needed to effectively quantify the dynamic evolution of device status from continuous image streams to distinguish between normal fluctuations and abnormal trends. Simultaneously, asynchronous environmental sensing data needs to be transformed into contextual information that can be temporally aligned with visual features and deeply integrated at the semantic level to accurately analyze the specific impact of environmental factors on device status. Summary of the Invention

[0005] The purpose of this invention is to provide an artificial intelligence image detection method, system, and storage medium based on the Internet of Things, so as to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides an artificial intelligence image detection method based on the Internet of Things, the method comprising: Simultaneously receive image data streams and environmental parameter data streams about the target device from the IoT sensor network deployed at the monitoring site. The image data streams contain several frames of raw detection images arranged in chronological order. The image data stream is processed by time-series slicing to obtain current image slices and past image slices with the same time span. Multi-scale feature pyramid extraction is performed on the current image slice to obtain deep visual features at multiple levels from coarse to fine. The environmental parameter data stream is decomposed into waveforms to separate periodic components, trend components, and random components. Based on the timestamp of the current image slice, the corresponding environmental state segments are extracted from the periodic components and the trend components. The deep visual features of the past image slices and the current image slices are differentially calculated to generate a dynamic difference spectrum representing changes in device state. At the same time, the deep visual features are coupled and encoded with the environmental state segments to generate an enhanced visual representation containing environmental context. Using the dynamic difference spectrum and the enhanced visual representation as dual inputs, a cascaded diagnostic inference engine is driven to work. The diagnostic inference engine sequentially performs local anomaly localization, global pattern matching, and defect type inference, and finally outputs image detection conclusions about the target device.

[0007] Preferably, the step of performing multi-scale feature pyramid extraction on the current image slice to obtain deep visual features at multiple levels from coarse to fine includes: At the initial level, a convolution operation with a large kernel size is performed on the current image slice to obtain global contour features covering the overall structure of the target device. In the intermediate layer, the global contour features are downsampled and convolutional operations with appropriately sized kernels are performed to obtain the regional structural features that characterize the main components of the target device. At the final level, the regional structural features are further downsampled and convolutional operations with smaller kernel sizes are performed to obtain local texture features that characterize the surface details of the target device. The global contour features, the regional structure features, and the local texture features are stacked in hierarchical order to form the multiple levels of deep visual features.

[0008] Preferably, the step of performing difference calculation on the deep visual features of the past image slices and the current image slices to generate a dynamic difference spectrum characterizing changes in device state includes: Extract the historical local texture features corresponding to the past image slices at the final level; Calculate the difference in feature values ​​between the local texture features of the current image slice and the historical local texture features at the corresponding spatial locations; The absolute values ​​of the eigenvalue differences at all spatial locations are summed and normalized to generate a two-dimensional difference intensity distribution map. The difference intensity distribution map is spatially aligned and superimposed with the global contour features of the current image slice to form the dynamic difference spectrum.

[0009] Preferably, the step of coupling and encoding the deep visual features with the environmental state fragment to generate an enhanced visual representation containing the environmental context includes: The periodic component values ​​and trend component values ​​in the environmental state segment are mapped into environmental feature vectors through a fully connected layer, respectively. The feature map of each level in the deep visual features is concatenated with the environmental feature vector in the channel dimension; A one-to-one convolution kernel is used to perform channel fusion and dimensionality reduction on the concatenated feature map, so that the features at each spatial location contain a mixed encoding of visual information and environmental information from the corresponding level. The feature maps of each level after channel fusion and dimensionality reduction are resampled to the size of the original input image and added element by element to synthesize the enhanced visual representation.

[0010] Preferably, the diagnostic reasoning engine sequentially performs local anomaly localization, global pattern matching, and defect type inference, including: During the local anomaly localization stage, the diagnostic inference engine inputs the dynamic difference spectrum into an encoder and outputs a heatmap, in which the highlighted areas indicate the locations on the target device where anomalies may occur. During the global pattern matching phase, the diagnostic reasoning engine performs a similarity measurement between the enhanced visual representation and a pre-stored normal device template library to identify the degree of deviation of the overall structure or main components of the target device from the normal pattern. During the defect type inference stage, the diagnostic reasoning engine performs a traversal query in a defect knowledge graph based on the abnormal location indicated by the heatmap and the degree of deviation. The defect knowledge graph defines the correspondence rules between different abnormal patterns and defect categories, thereby inferring the image detection conclusion.

[0011] Preferably, the construction and updating of the pre-stored normal device template library includes: During the fault-free operation of the target device, its standard image data stream and environmental parameter data stream are continuously acquired; The standard image data stream is subjected to the multi-scale feature pyramid extraction to obtain a standard deep visual feature set; The environmental parameter data stream is decomposed into waveforms, and a standard environmental state segment aligned with the timestamp of the standard image data stream is extracted. Using the same coupling coding method as that used to generate the enhanced visual representation, the standard deep visual feature set and the standard environmental state fragments are processed to generate a series of standard enhanced visual representations, which are then stored in the normal device template library.

[0012] Preferably, in the defect type inference stage, the diagnostic reasoning engine performs a traversal query in a defect knowledge graph based on the abnormal location indicated by the heatmap and the degree of deviation, including: The coordinate information of the highlighted areas in the heat map is converted into corresponding component identifiers on the 3D model of the target device; The degree of deviation is quantified into a deviation score, which reflects the overall magnitude of the difference between the target device and the normal mode. Using the component identifier and the deviation score as joint query conditions, a matching path is searched in the relational network formed by the nodes and edges of the defect knowledge graph. The label carried by the endpoint node of the path is the inferred defect type.

[0013] Preferably, the method of using a 1x1 convolution kernel to perform channel fusion and dimensionality reduction on the concatenated feature map ensures that the features at each spatial location contain a hybrid encoding of visual information and environmental information from the corresponding level, including: The extended feature map formed by concatenating the feature map of each level in the deep visual features with the environmental feature vector is used as input; The extended feature map is convolved using a one-by-one convolutional layer with the same number of convolutional kernels as the number of channels in the extended feature map. By combining the linear elements of the 1x1 convolutional layers with a nonlinear activation function, the channel dimension of the expanded feature map is reduced to the preset target number of channels. After channel fusion and dimensionality reduction, the feature vector at each spatial location of the output feature map integrates the original visual feature components and the mapped environmental feature components, forming the hybrid encoding.

[0014] Preferably, the present invention also includes an IoT-based artificial intelligence image detection system for implementing the IoT-based artificial intelligence image detection method described above, the system comprising: The data synchronization receiving module is used to synchronously receive image data streams and environmental parameter data streams about the target device from the IoT sensor network deployed at the monitoring site; The temporal slicing and feature extraction module is used to perform temporal slicing processing on the image data stream to obtain current image slices and past image slices with the same time span, and to perform multi-scale feature pyramid extraction on the current image slice to obtain deep visual features at multiple levels from coarse to fine. The environmental data decomposition module is used to perform waveform decomposition on the environmental parameter data stream, separate periodic components, trend components and random components, and extract corresponding environmental state segments from the periodic components and trend components based on the timestamp of the current image slice. The feature fusion processing module is used to perform differential calculation on the deep visual features of the past image slice and the current image slice to generate a dynamic difference spectrum representing the changes in device state. At the same time, the deep visual features are coupled and encoded with the environmental state segment to generate an enhanced visual representation containing the environmental context. The cascaded diagnostic reasoning module is used to drive the cascaded diagnostic reasoning engine to work with the dynamic difference spectrum and the enhanced visual representation as dual inputs. The diagnostic reasoning engine sequentially performs local anomaly localization, global pattern matching and defect type inference, and finally outputs the image detection conclusion about the target device.

[0015] Preferably, the present invention further includes a computer-readable storage medium comprising a stored computer program, wherein when the device containing the computer-readable storage medium executes the computer program, it implements the above-described AI image detection method based on the Internet of Things.

[0016] Compared with the prior art, the beneficial effects of the present invention are: The image data stream is processed by temporal slicing to obtain current and past image slices. The extracted deep visual features are then differentially calculated to generate a dynamic difference spectrum. This directly quantifies the gradient and pattern of changes in the device's visual state over time at the deep feature level, enabling the system to sensitively capture gradual wear, slow structural deformation, or periodically occurring weak anomaly signals that are overlooked by traditional single-frame analysis. This feature-difference-based dynamic representation filters out static interference such as background and lighting, focusing the analysis on the "change" itself, thus improving the detection capability and early warning timeliness of gradual changes in device state and early weak faults.

[0017] The environmental parameter data stream is decomposed into waveforms to separate its inherent periodic and trend components. Corresponding environmental state segments are then extracted based on image timestamps and coupled with visual features for encoding. Waveform decomposition removes high-frequency random noise from the environmental signal, preserving the slowly changing trends and regular periodic components with clear physical meaning. Precise extraction based on timestamps ensures strict temporal synchronization between the environmental context and visual information. The coupling encoding process achieves deep fusion of environmental state segments and deep visual features at a high-level semantic level, enabling the generated enhanced visual representation to inherently contain the correlation information of "under what environmental trend or periodic phase this visual feature was observed." This allows the diagnostic model to learn the complex mapping relationship between environmental variables and equipment performance, enabling more accurate assessment of equipment status when facing diurnal temperature variations, seasonal load changes, or specific operating cycles, reducing misjudgments caused by environmental factors. Attached Figure Description

[0018] Figure 1 This is a schematic diagram illustrating the working principle of the IoT-based artificial intelligence image detection method described in this invention. Figure 2 The flowchart for multi-scale feature pyramid extraction; Figure 3 A flowchart for generating dynamic difference spectra; Figure 4 A comparison chart of template matching scores in the global pattern matching stage of IoT AI image detection; Figure 5 A comparison chart of deviation scores for the defect type inference stage of IoT AI image detection. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Please see Figure 1 This invention provides an AI-based image detection method based on the Internet of Things (IoT). The method includes an integrated processing framework designed to collaboratively analyze visual and environmental data from a monitoring site. The implementation begins with a data synchronization reception stage, simultaneously acquiring image data streams and environmental parameter data streams from an IoT sensor network deployed at the target device site. The image data streams consist of consecutive frames of raw detection images arranged chronologically. Subsequently, the system performs a temporal slicing operation on the consecutive image data streams, generating two image data units with consistent time spans: a current image slice representing the recent state and a past image slice representing the historical state. For the current image slice, a multi-scale feature pyramid extraction process is performed to obtain multiple levels of deep visual features, from macroscopic overall structure to microscopic surface details. Simultaneously, the environmental parameter data stream is fed into a waveform decomposition module, decomposed into periodic components, trend components, and random components with clear physical meaning, and based on the timestamp information carried by the current image slice, environmental state fragments associated with the context of that time point are extracted from the periodic and trend components. In the feature fusion stage, on the one hand, deep visual features from past and current image slices are used for differential calculation, generating a dynamic difference spectrum that intuitively reflects the changes in device status over time. On the other hand, deep visual features and environmental state fragments are fused through coupling coding technology to generate an enhanced visual representation containing environmental condition information. Finally, the dynamic difference spectrum and the enhanced visual representation serve as two parallel input paths, jointly driving a cascaded diagnostic inference engine. This engine operates sequentially according to a preset logical order: first, local anomaly localization is performed; then, global pattern matching is executed; finally, defect type inference is completed, and image detection conclusions regarding the target device status and potential problems are output.

[0021] In one embodiment of the present invention, see [reference] Figure 2In this embodiment, at the initial level, a large-sized convolutional kernel is used to convolve the input current image slice. This operation aims to capture the overall shape and general outline of the target device in the image, forming global contour features. At the intermediate level, the global contour features output from the previous level are downsampled to reduce spatial resolution, and further convolution is performed using a moderately sized convolutional kernel. This process focuses on identifying and extracting the shape and structural information of the main components or key regions inside the target device, obtaining regional structural features. At the final level, the regional structural features output from the intermediate level are downsampled again, and fine convolution is performed using a smaller-sized convolutional kernel. This level focuses on resolving subtle textures, blemishes, or local features on the device surface, obtaining local texture features. After completing the feature extraction at these three levels, the obtained global contour features, regional structural features, and local texture features are stacked and organized according to their extraction hierarchy—that is, from coarse to fine in terms of feature abstraction—to form a complete set of deep visual features containing multiple levels.

[0022] In practice, multi-scale feature pyramid extraction involves layer-by-layer convolution and downsampling processing of the current image slice, which refers to a set of image frames within a fixed time window extracted from a continuous image data stream. Processing of the current image slice begins at the initial level, where convolution operations with larger kernel sizes are used to extract features. Larger kernel sizes cover a wider receptive field in the image, thus capturing the overall shape boundary and macroscopic structural layout of the target device within the frame. The output of this step is defined as global contour features. These global contour features carry the main shape information that distinguishes the target device from the background and other objects.

[0023] In some embodiments, the output of the initial layer, i.e., the global contour features, is passed to an intermediate layer for processing. At the intermediate layer, the received global contour features are first downsampled to reduce their spatial dimensionality. Then, a convolution operation with a suitable kernel size is applied to perform secondary feature extraction on the downsampled feature map. This convolution operation balances the receptive field size with the ability to preserve feature details, allowing this layer to focus on analyzing the regional shapes of key functional components within the target equipment, such as engines, tanks, and conveyor belts, and the structural relationships between them. The resulting features are defined as regional structural features. These regional structural features reflect the morphological information of the key components of the equipment.

[0024] It is understandable that the processing results of the intermediate level, namely the regional structural features, will be further input to the final level. In the final level, the regional structural features first undergo another downsampling process to achieve further compression and abstraction of the features. Then, a convolution operation with a small kernel size is used to perform a high-precision scan of the feature map. The convolution operation with a small kernel size focuses on subtle pattern changes in the neighborhood of local pixels, thereby effectively extracting local texture features that characterize whether there are scratches, rust, oil stains, cracks or abnormal temperature distribution on the surface of the target device.

[0025] Optionally, the convolution operations at all three levels can be accompanied by non-linear activation functions and batch normalization to enhance the expressive power of the features. For example, a modified linear unit function can be applied after the convolution operation. After completing the feature extraction at the final level, the system organizes the obtained feature set, stacking and integrating global contour features, region structure features, and local texture features according to their extraction hierarchy, i.e., from coarse to fine abstraction. The stacking of global contour features, region structure features, and local texture features constitutes a deep visual feature pyramid with multi-level semantic information. The construction formula of the deep visual feature pyramid can be expressed as: in: This represents the final, stacked pyramid of deep visual features. This represents a stacking operation performed in hierarchical order. This represents the global contour features extracted from the initial layer. This represents the regional structural features extracted from the intermediate level. This represents the local texture features extracted from the final layer.

[0026] In some embodiments, the convolutional operations of the initial, intermediate, and final layers can be implemented based on different stages of a shared backbone network. For example, specific stage outputs of a deep residual network or visual transformer architecture can be used as global contour features, region structure features, and local texture features, respectively. Optionally, the features extracted from each layer can be subjected to independent channel-adjusting convolutions before stacking to unify the channel dimensions, facilitating subsequent feature fusion and processing.

[0027] In one embodiment of the present invention, see [reference] Figure 3In this embodiment, historical local texture features, corresponding to past image slices at the final level, are extracted. Next, for the local texture features extracted from the current image slice at the final level, element-wise calculations are performed between these features and the corresponding historical local texture features at the same spatial pixel or feature map location to obtain the feature value difference at each location. Then, the absolute values ​​of the feature value differences calculated for all spatial locations are taken and summed over the entire spatial dimension. The summation result is then mapped to a standardized numerical range to generate a two-dimensional difference intensity distribution map. The value of each point in this map represents the temporal intensity of change at the corresponding spatial location. Finally, the generated difference intensity distribution map is aligned spatially with the global contour features extracted from the current image slice at the initial level, and the two are superimposed or fused at their respective spatial locations to form the final dynamic difference spectrum. This spectrum simultaneously contains information about the location of the change and the overall contour background of the device.

[0028] In practice, the process of generating a dynamic difference spectrum begins with feature extraction from past image slices. Past image slices refer to a set of images within an equal-length time window that precedes the current image slice in the time series. Features corresponding to the final level are extracted from past image slices; these features are called historical local texture features. The method for extracting historical local texture features is completely consistent with the method for extracting local texture features from the current image slice, ensuring alignment and comparability of the feature space.

[0029] In some embodiments, after obtaining the local texture features and historical local texture features of the current image slice, calculations need to be performed at corresponding positions on the same spatial coordinate grid to calculate the feature value difference between the local texture features and historical local texture features at the corresponding spatial positions. The feature value difference directly reflects the numerical change of the surface texture features at the same device location between two time points. For a feature map with dimensions H, W, and C channels, the feature value difference calculation at spatial position (i,j) involves the feature vectors of all channels at that position.

[0030] It is understandable that after calculating the eigenvalue differences for all spatial locations, it is necessary to integrate and visualize the overall change intensity. This involves summing the absolute values ​​of the eigenvalue differences for all spatial locations and normalizing the mapping. The absolute value summation operation is performed on all channel differences for each spatial location, resulting in a two-dimensional difference intensity matrix. The normalization mapping linearly transforms the values ​​in this matrix to a preset range, generating a two-dimensional difference intensity distribution map. The brightness or numerical value of each pixel in the difference intensity distribution map visually represents the degree of drastic change in the temporal state of the corresponding area on the surface of the target device.

[0031] Optionally, the generation of the difference intensity distribution map can follow a specific mathematical expression, for example, the difference intensity distribution map. The result is obtained through calculation using the formula: in: This represents the intensity value of the difference intensity distribution map at location (i,j). Represents the normalized mapping function. It represents the total number of channels for local texture features. It represents the feature value of the local texture features of the current image slice at position (i,j) and channel c. These are the feature values ​​of historical local texture features at the same location and in the same channel. Summation and absolute value operations are performed on the channel dimension c.

[0032] In some embodiments, after constructing the difference intensity distribution map, it needs to be combined with the device's spatial contour information. This involves spatially aligning and overlaying the difference intensity distribution map with the global contour features of the current image slice. Since the difference intensity distribution map and the global contour features originate from different feature levels and have different spatial resolutions, the spatial alignment operation typically involves upsampling the difference intensity distribution map or downsampling the global contour features to ensure they have the same height and width. The overlay operation can be an element-wise weighted sum or channel stitching to form the final dynamic difference spectrum. Optionally, the spatial alignment process can utilize bilinear interpolation to upsample the difference intensity distribution map. During the overlay operation, different weight coefficients can be assigned to the difference intensity distribution map and the global contour features to adjust the saliency of the change information and the contour background in the dynamic difference spectrum.

[0033] In one embodiment of the present invention, the periodic component numerical sequence and the trend component numerical sequence contained in the environmental state segment are respectively input into two independent mapping networks composed of fully connected layers, and converted into fixed-dimensional environmental feature vectors. For the feature map of each level in the deep visual features, such as global contour features, regional structure features, and local texture features, they are concatenated with the environmental feature vector in the channel dimension, thereby expanding the number of feature channels at each spatial location to form an expanded feature map. The expanded feature map obtained after concatenation is processed using a 1x1 convolutional kernel, which is configured with the same number of convolutional kernels as the number of channels in the expanded feature map. Through the linear combination operation of this convolutional layer and subsequent nonlinear activation function processing, the high channel dimension of the expanded feature map is compressed to a preset, lower target number of channels, realizing channel fusion and dimensionality reduction. After this operation, the feature vector at each spatial location of the output feature map is formed by fusing the original visual feature components and the mapped environmental feature components, forming a hybrid encoding that includes the environmental context. Finally, the feature maps of each level after channel fusion and dimensionality reduction are restored to the same spatial size as the original input image through upsampling, and the feature maps of all levels are added element by element to synthesize the final enhanced visual representation.

[0034] The coupled encoding process begins with the generation of environmental feature vectors. The periodic and trend component numerical sequences from the environmental state segments are taken as independent inputs and subjected to linear transformation and nonlinear mapping respectively through two structurally consistent fully connected layers. The fully connected layers map the one-dimensional numerical sequences into fixed-length environmental feature vectors. These environmental feature vectors encapsulate information about periodic environmental fluctuations and long-term environmental evolution trends related to the temporal context of the current image slice. In some embodiments, after mapping the environmental feature vectors, the system begins processing each independent level of the deep visual features. Deep visual features comprise multiple levels of feature maps, from coarse to fine, such as global contour features, region structure features, and local texture features. For each level of feature map in the deep visual features, the feature map is concatenated with the environmental feature vector along the channel dimension. This concatenation operation broadcasts and copies the environmental feature vector to every spatial location of the feature map, then connects it to the original visual feature channels to form an expanded feature map with increased channel count. The expanded feature map simultaneously contains a region visual descriptor from the image and a state descriptor from the environment at the same spatial location.

[0035] It's understandable that the expanded feature map significantly increases the channel dimension, leading to computational redundancy and overfitting if used directly. Therefore, a 1x1 convolution kernel is needed to perform channel fusion and dimensionality reduction on the concatenated feature map. A 1x1 convolution kernel refers to a convolution operation where the kernel size is 1x1 in spatial dimension. A 1x1 convolution kernel focuses on combining information across channels. Taking the expanded feature map of each level in the deep visual feature map as input, a 1x1 convolution layer with the same number of kernels as the number of channels in the expanded feature map is used to perform convolution operations on the expanded feature map. Each kernel of the 1x1 convolution layer corresponds to a target channel in the output feature map.

[0036] In some embodiments, the operation of a 1x1 convolutional layer achieves feature compression and re-encoding. Through linear combination of 1x1 convolutional layers and processing with a non-linear activation function, the channel dimension of the expanded feature map is reduced to a preset target number of channels. The preset target number of channels is a fixed value smaller than the number of channels in the expanded feature map. The dimensionality-reduced output feature map maintains the original spatial resolution, but the feature vector at each spatial location has been recalculated. After channel fusion and dimensionality reduction, the feature vector at each spatial location of the output feature map integrates the original visual feature components and the mapped environmental feature components, forming a hybrid encoding. The hybrid encoding is the basic unit for subsequent synthesis of enhanced visual representations.

[0037] Optionally, the mathematical process of channel fusion and dimensionality reduction can be described as follows: in: This represents the hybrid encoded feature map after the l-th layer has undergone a one-to-one convolution process. Represents a non-linear activation function. It is the total number of channels in the extended feature map. This represents the convolution weight vector corresponding to the k-th input channel in a 1x1 convolutional layer. This represents the convolution operation. This represents the original deep visual feature map of level l. Represents environmental feature vectors. This represents the concatenation operation along the channel dimension. The summation operation is performed on all input channels k, with the weight vector... Its dimensions are one by one.

[0038] It is understandable that after obtaining the hybrid encoded feature maps at each level, they need to be integrated into a unified enhanced visual representation. This involves resampling the feature maps from each level, after channel fusion and dimensionality reduction, to the size of the original input image and then adding them element-wise. Upsampling typically employs deconvolution or bilinear interpolation to enlarge the hybrid encoded feature maps of different spatial resolutions to the same height and width as the original input image. The element-wise addition operation adds the values ​​of all upsampled feature maps at their corresponding spatial locations and channels, synthesizing the final enhanced visual representation. This enhanced visual representation is spatially aligned with the input image and incorporates multi-scale visual details and synchronous environmental context in its feature dimension. Optionally, the upsampling process can be performed layer by layer. For example, the coarsest level feature map can be upsampled to the size of an adjacent finer level, added to the finer level feature map, and then upsampled again until the target size is reached. Before element-wise addition, learnable weights can be applied to the feature maps of different levels to adjust the contribution ratio of each level.

[0039] In one embodiment of the present invention, during the local anomaly localization stage, the diagnostic inference engine inputs a dynamic difference spectrum into a neural network with an encoder structure. The encoder outputs a heatmap corresponding to the input size. Regions activated or with high values ​​in the heatmap correspond to locations on the target device image where significant state changes or anomalies have occurred. During the global pattern matching stage, the diagnostic inference engine performs similarity measurement calculations between the enhanced visual representation and multiple templates in a pre-generated and stored normal device template library. By comparison, it identifies the overall structural form of the current target device or the appearance pattern of its main components, and the similarity or deviation degree between these patterns and standard normal patterns. During the defect type inference stage, the diagnostic inference engine integrates the results of the first two stages—the anomaly location information indicated by the heatmap and the calculated overall deviation degree—and traverses and queries a pre-constructed defect knowledge graph. This defect knowledge graph defines the association rules between different anomaly spatial patterns, deviation ranges, and specific defect categories using a graphical structure. By matching the queried paths, the defect type is inferred as the image detection conclusion.

[0040] In its implementation, the cascaded inference of the diagnostic inference engine unfolds based on two parallel input paths: dynamic difference spectrum and enhanced visual representation. The local anomaly localization stage receives the dynamic difference spectrum as input and feeds it into a neural network with an encoder structure. The encoder typically consists of multiple convolutional and pooling layers, used for feature abstraction and spatial compression of the dynamic difference spectrum. The encoder's final output layer generates a single-channel heatmap corresponding to the spatial size of the input dynamic difference spectrum through a one-to-one convolution operation and an activation function. The values ​​at each pixel location in the heatmap are normalized, and areas with higher values ​​are highlighted. These highlighted areas spatially correspond to locations on the target device image where significant state changes or anomalies have occurred.

[0041] In some embodiments, the global pattern matching stage immediately follows the local anomaly localization stage. The diagnostic inference engine performs a similarity measurement between the enhanced visual representation and a pre-generated and stored normal device template library. The normal device template library stores several standard enhanced visual representations collected and generated under various typical environmental parameters in a fault-free state of the target device as templates. The similarity measurement calculates the matching score between the currently input enhanced visual representation and each template in the template library. The matching score reflects the distance or similarity in the feature space. By comparing all matching scores, the overall structural form of the current target device or the appearance pattern of its main components, and the overall deviation from the standard normal pattern, are identified. It can be understood that the defect type inference stage integrates the output results of the previous two stages to make the final decision. The diagnostic inference engine traverses and queries a predefined defect knowledge graph based on the anomaly location indicated by the heatmap and the calculated deviation. The defect knowledge graph stores domain knowledge in a graphical structure. The nodes of the defect knowledge graph represent different device components, observable anomaly patterns, combinations of environmental conditions, or specific defect categories. The edges of the defect knowledge graph define the logical associations, causal relationships, or conditional probabilities between nodes.

[0042] Optionally, the similarity measure can be calculated based on cosine similarity, and its formula is expressed as: in: Represents the matching score. The feature vector obtained after flattening the enhanced visual representation of the input. This represents the feature vector of a specific template in the normal device template library. Representing vectors with vector dot product operation, Representing vectors The length of the mold, Representing vectors Modulus length. Matching score. The value ranges from -1 to 1, with higher values ​​indicating higher similarity. See Table 1.

[0043] Table 1: Score Indicator of Enhanced Visual Representation and Template Library Matching Template Number Template Description Match score (S) Template A Standard working condition template 0.92 Template B High-temperature working condition template 0.87 Template C High vibration working condition formwork 0.45 In some embodiments, the global pattern matching stage can determine the degree of deviation by comparing the distribution of the current matching score with that of historical matching scores. The traversal query of the defect knowledge graph can be viewed as a rule-based reasoning process. For example, if the highlighted area of ​​the heatmap maps to the "bearing housing" component, and the global matching score is below a threshold θ, then the defect knowledge graph is traversed along the path "bearing housing" -> "abnormal vibration" -> "poor lubrication," with the endpoint "poor lubrication" as the inferred defect type output. The defect knowledge graph allows the diagnostic process to go beyond simple pattern classification, incorporating joint reasoning based on location, degree, and domain logic. Optionally, the encoder used in the local anomaly localization stage can be a lightweight convolutional neural network, such as the encoding part of U-Net, whose output heatmap can be binarized by setting an activation threshold, thereby clearly defining the abnormal region. It is understood that the templates in the normal device template library can be clustered and indexed based on environmental state fragments to accelerate the retrieval speed of the global pattern matching stage.

[0044] See Figure 4 This is a template matching score comparison chart from the global pattern matching stage of IoT AI image detection, used to evaluate the similarity between the current state of the target device and the template for normal operating conditions. The similarity threshold is 0.7; values ​​below this indicate that the device state deviates from the normal pattern. The data shows a significant difference between the current state of the device and the normal template for high-vibration conditions. Combined with the results of local anomaly localization, the failure risk of vibration-related components (such as bearings) of the device can be prioritized. This type of chart is used in the pattern matching stage of device anomaly detection. By comparing the similarity between the current device and normal templates for multiple operating conditions, deviations in the device state (such as high vibration) can be quickly located, providing a basis for subsequent defect reasoning.

[0045] In one embodiment of the present invention, the construction of the normal equipment template library begins when the target equipment is in a known fault-free operating cycle, during which its standard image data stream and corresponding environmental parameter data stream are continuously collected. For the collected standard image data stream, the same multi-scale feature pyramid extraction method used for processing the current image slice is employed to obtain a series of standard deep visual features. Simultaneously, waveform decomposition is performed on the environmental parameter data stream, and environmental state segments precisely aligned with the timestamps of each frame of the standard image data stream are extracted. Using the same coupling coding method as for generating enhanced visual representations, each set of standard deep visual features and its corresponding environmental state segments are processed to generate a series of standard enhanced visual representations, which constitute the storage content of the normal equipment template library. During the defect knowledge graph query process, the image coordinates of the highlighted areas in the heatmap generated during the local anomaly localization stage are converted into specific component identifiers on the target equipment's 3D model through a pre-established mapping relationship. Simultaneously, the deviation calculated during the global pattern matching stage is quantified into a specific deviation score. Using component identifiers and deviation scores as a set of joint query conditions, the system performs retrieval and matching in a defect knowledge graph relationship network composed of nodes and edges. Nodes in the graph represent equipment components, abnormal phenomena, or defect types, and edges represent the logical relationships between them. The system traverses along the matching path until it reaches the endpoint node, and the label carried by that node is inferred as the final defect type.

[0046] In practical implementation, the construction and updating of a pre-stored normal device template library is fundamental to ensuring the accuracy of global pattern matching. The construction of the normal device template library begins during a defined period of fault-free device operation. During the fault-free operation of the target device, the system continuously collects image data streams and environmental parameter data streams about the target device. The collected image data streams are defined as standard image data streams, and the environmental parameter data streams are defined as standard environmental parameter data streams. A multi-scale feature pyramid extraction process, identical to that in the online detection phase, is performed on the standard image data streams to obtain a set covering different time points and different operating states. This feature set is called the standard deep visual feature set, which contains the multi-scale visual feature representation of the device under normal conditions, from global to local.

[0047] In some embodiments, in sync with the acquired standard image data stream, a complete waveform decomposition process is performed on the environmental parameter data stream, decomposing it into periodic components, trend components, and random components. Based on the precise timestamp carried by each frame of the standard image data stream, data segments perfectly aligned with the timestamps are extracted from the decomposed periodic and trend components; these data segments are called standard environmental state segments. Each standard environmental state segment corresponds one-to-one with each feature sample in the standard deep visual feature set in the time dimension. Using the same coupling coding method as the online generation of enhanced visual representations, each pair of paired standard deep visual features and standard environmental state segments is fused to generate a series of standard enhanced visual representations. Each standard enhanced visual representation represents a health state template of the device under specific environmental conditions. All generated standard enhanced visual representations are systematically stored in a normal device template library for persistent storage.

[0048] It is understandable that the normal equipment template library needs an update mechanism to adapt to the slow aging or operational drift of equipment. The update process can be achieved by periodically re-collecting fault-free data and repeating the aforementioned construction process, or by incrementally learning new samples of normal conditions, confirmed by manual verification, and their generated standard enhanced visual representations, to be added to the normal equipment template library. In the defect type inference stage, the specific implementation of the diagnostic inference engine's graph query based on the abnormal location and deviation indicated by the heatmap involves coordinate transformation and quantization processes. This process converts the image coordinate information of the highlighted areas in the heatmap into corresponding component identifiers on the target equipment's 3D model. The transformation process relies on a pre-generated coordinate mapping relationship, which establishes a correspondence between the image pixel coordinate system and the component coordinate system of the equipment's 3D model. The component identifier is used to locate the starting node for the query in the defect knowledge graph.

[0049] Optionally, the quantification process of deviation follows a clear mathematical rule, quantifying the degree of deviation into a deviation score. The quantification formula can be expressed as: in: This represents the calculated deviation score. This represents the total number of templates in the normal equipment template library. This represents the matching score obtained by measuring the similarity between the current enhanced visual representation and the 1st to Nth templates in the normal device template library. This function selects the maximum value among all matching scores. Deviation score The value range is between 0 and 1, deviating from the fraction. The higher the value, the greater the overall difference between the target device and the normal mode.

[0050] In some embodiments, a traversal search is performed on the relational network of nodes and edges in the defect knowledge graph, using component identifier and deviation score as joint query conditions. The relational network of the defect knowledge graph is stored in a graph structure, where nodes represent equipment components, abnormal phenomena, environmental conditions, or defect types, and edges represent causal relationships, conditional probabilities, or logical implications between nodes. The traversal query finds the starting node that matches the component identifier and moves along the edges to adjacent nodes according to the size range of the deviation score until a terminal node that cannot be extended further is reached. The label carried by this terminal node is the specific defect type inferred by the system. The construction of the defect knowledge graph is based on the prior knowledge of domain experts and the summary of historical failure cases. It can be understood that the coordinate mapping relationship can be established by rendering the 3D model of the equipment from multiple perspectives and registering it with actual captured images. In specific implementations, the construction of the 3D model of the equipment is based on the computer-aided design data of the target equipment or the geometric model obtained through 3D scanning technology. This model accurately represents the physical structure of the equipment and the spatial relationships of its components. The multi-view rendering process uses a graphics rendering engine to virtually image the 3D model of the equipment from multiple preset perspectives, generating a series of synthetic images corresponding to different observation angles. The actual captured images come from image data streams collected by an IoT sensor network deployed on-site, with timestamps aligned to the current analysis time. The registration operation uses feature point matching algorithms, such as scale-invariant feature transformation or accelerated robust feature transformation, to spatially align the synthesized image with the actual captured image, calculating affine or perspective transformation parameters between the two to establish a precise mapping between the image pixel coordinate system and the coordinate system of the device's 3D model components. The deviation score can also be defined based on the statistical difference between the current matching score and the historical matching score distributions.

[0051] See Figure 5 This is a deviation score comparison chart from the defect type inference stage of IoT AI image detection, used to correlate equipment component anomalies with defect types. The defect judgment threshold is 0.7; values ​​above this indicate an anomaly in the corresponding component. The data shows a strong correlation between bearing housing anomalies and "poor lubrication, bearing wear," and a strong correlation between motor end cover anomalies and "loose bolts." Subsequent maintenance should prioritize checking and repairing these components. This type of chart is used in the equipment defect reasoning stage. By matching the deviation score of component anomalies with defect types, it quickly locates faulty components and specific defects, providing accurate data for equipment maintenance.

[0052] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0053] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An artificial intelligence image detection method based on Internet of Things, characterized in that, Comprise: synchronously receiving image data stream and environmental parameter data stream about target equipment from Internet of Things sensor network deployed in monitoring site, the image data stream contains several frames of original detection images arranged in time sequence; performing time sequence slicing processing on the image data stream to obtain current image slice and past image slice with same time span, and performing multi-scale feature pyramid extraction on the current image slice to obtain deep visual features of multiple levels from coarse to fine; performing waveform decomposition on the environmental parameter data stream to separate periodic component, trend component and random component, and according to the timestamp of the current image slice, intercepting corresponding environmental state segment from the periodic component and the trend component; performing difference calculation on the deep visual features of the past image slice and the current image slice to generate dynamic difference spectrum representing equipment state change, and coupling and encoding the deep visual features and the environmental state segment to generate enhanced visual representation containing environmental context; taking the dynamic difference spectrum and the enhanced visual representation as double-path input to drive the cascaded diagnostic reasoning engine to work, the diagnostic reasoning engine sequentially performs local anomaly positioning, global pattern matching and defect type inference, and finally outputs image detection conclusion about the target equipment. 2.The Internet of Things based artificial intelligence image detection method according to claim 1, characterized in that, The multi-scale feature pyramid extraction on the current image slice to obtain deep visual features of multiple levels from coarse to fine comprises: in the initial level, performing convolution operation with large convolution kernel size on the current image slice to obtain global outline feature covering the overall structure of the target equipment; in the middle level, performing down-sampling and convolution operation with moderate convolution kernel size on the global outline feature to obtain regional structure feature representing main components of the target equipment; in the final level, performing further down-sampling and convolution operation with small convolution kernel size on the regional structure feature to obtain local texture feature representing surface details of the target equipment; stacking the global outline feature, the regional structure feature and the local texture feature in hierarchical order to form the deep visual features of multiple levels. 3.The Internet of Things based artificial intelligence image detection method according to claim 2, characterized in that, The difference calculation on the deep visual features of the past image slice and the current image slice to generate dynamic difference spectrum representing equipment state change comprises: extracting historical local texture feature corresponding to the final level of the past image slice respectively; calculating feature value difference of the local texture feature of the current image slice and the historical local texture feature in corresponding spatial position; performing absolute value summation and normalization mapping on feature value difference of all spatial positions to generate a two-dimensional difference intensity distribution map; performing spatial alignment and superposition of the difference intensity distribution map and the global outline feature of the current image slice to form the dynamic difference spectrum. 4.The Internet of Things based artificial intelligence image detection method according to claim 1, characterized in that, The coupling and encoding of the deep visual features and the environmental state segment to generate enhanced visual representation containing environmental context comprises: mapping periodic component value and trend component value in the environmental state segment into environmental feature vector through full connection layer respectively; concatenate the feature maps of each level in the deep visual features with the environmental feature vector in the channel dimension; perform channel fusion and dimension reduction on the concatenated feature maps using a one-by-one convolution kernel, so that the features of each spatial position contain mixed encoding of visual information and environmental information from the corresponding level; re-sample the feature maps of each level after channel fusion and dimension reduction to the size of the original input image and perform element-wise addition to synthesize the enhanced visual representation. 5.The Internet of Things based artificial intelligence image detection method according to claim 1, characterized in that, The diagnostic reasoning engine sequentially performs local anomaly positioning, global pattern matching, and defect type inference, including: In the local anomaly positioning stage, the diagnostic reasoning engine inputs the dynamic difference spectrum into an encoder and outputs a heat map, wherein the highlighted area in the heat map indicates the position where the anomaly on the target device is likely to occur; In the global pattern matching stage, the diagnostic reasoning engine performs similarity measurement between the enhanced visual representation and a pre-stored normal device template library to identify the deviation of the overall structure or main components of the target device relative to the normal pattern; In the defect type inference stage, the diagnostic reasoning engine performs traversal query in a defect knowledge graph based on the abnormal position indicated by the heat map and the deviation, wherein the defect knowledge graph defines the corresponding rules between different abnormal patterns and defect categories, thereby inferring the image detection conclusion. 6.The Internet of Things based artificial intelligence image detection method according to claim 5, characterized in that, The construction and update of the pre-stored normal device template library include: During the fault-free operation of the target device, continuously collect its standard image data stream and environmental parameter data stream; perform the multi-scale feature pyramid extraction on the standard image data stream to obtain a standard deep visual feature set; perform the waveform decomposition on the environmental parameter data stream and intercept a standard environmental state segment aligned with the timestamp of the standard image data stream; use the same coupling encoding method as generating the enhanced visual representation to process the standard deep visual feature set and the standard environmental state segment to generate a series of standard enhanced visual representations, and store them in the normal device template library. 7.The Internet of Things based artificial intelligence image detection method according to claim 5, characterized in that, In the defect type inference stage, the diagnostic reasoning engine performs traversal query in a defect knowledge graph based on the abnormal position indicated by the heat map and the deviation, including: convert the coordinate information of the highlighted area in the heat map into the corresponding component identifier on the three-dimensional model of the target device; quantify the deviation degree into a deviation score, which reflects the overall difference between the target device and the normal pattern; use the component identifier and the deviation score as joint query conditions to find a matching path in the relationship network composed of the nodes and edges of the defect knowledge graph, and the label carried by the end node of the path is the inferred defect type. 8.The Internet of Things based artificial intelligence image detection method of claim 4, wherein, The use of a one-by-one convolution kernel to perform channel fusion and dimension reduction on the concatenated feature maps so that the features of each spatial position contain mixed encoding of visual information and environmental information from the corresponding level includes: taking the extended feature map formed by concatenating the feature maps of each level in the deep visual features with the environmental feature vector as input; The extended feature map is convolved using a one-by-one convolutional layer with the same number of convolutional kernels as the number of channels in the extended feature map. By combining the linear elements of the 1x1 convolutional layers with a nonlinear activation function, the channel dimension of the expanded feature map is reduced to the preset target number of channels. After channel fusion and dimensionality reduction, the feature vector at each spatial location of the output feature map integrates the original visual feature components and the mapped environmental feature components, forming the hybrid encoding.

9. An artificial intelligence image detection system based on Internet of Things, characterized in that, The method for implementing the Internet of Things-based artificial intelligence image detection method as described in any one of claims 1-8 includes: The data synchronization receiving module is used to synchronously receive image data streams and environmental parameter data streams about the target device from the IoT sensor network deployed at the monitoring site; The temporal slicing and feature extraction module is used to perform temporal slicing processing on the image data stream to obtain current image slices and past image slices with the same time span, and to perform multi-scale feature pyramid extraction on the current image slice to obtain deep visual features at multiple levels from coarse to fine. The environmental data decomposition module is used to perform waveform decomposition on the environmental parameter data stream, separate periodic components, trend components and random components, and extract corresponding environmental state segments from the periodic components and trend components based on the timestamp of the current image slice. The feature fusion processing module is used to perform differential calculation on the deep visual features of the past image slice and the current image slice to generate a dynamic difference spectrum representing the changes in device state. At the same time, the deep visual features are coupled and encoded with the environmental state segment to generate an enhanced visual representation containing the environmental context. The cascaded diagnostic reasoning module is used to drive the cascaded diagnostic reasoning engine to work with the dynamic difference spectrum and the enhanced visual representation as dual inputs. The diagnostic reasoning engine sequentially performs local anomaly localization, global pattern matching and defect type inference, and finally outputs the image detection conclusion about the target device.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein when the device containing the computer-readable storage medium executes the computer program, it implements the Internet of Things-based artificial intelligence image detection method as described in any one of claims 1 to 8.

Images