Defect autonomous decision-making method, device and equipment for industrial inspection and medium

Abstract

The invention provides an industrial inspection-oriented defect autonomous decision-making method, device and equipment and a medium, and belongs to the technical field of intelligent manufacturing. The equipment defect autonomous decision-making method comprises the following steps: acquiring equipment operation state information of industrial equipment by using a multispectral fusion terminal, and uploading the equipment operation state information to a cloud server; a built-in cross-modal deep analysis model of the cloud server is used to carry out cross-modal learning on the equipment operation state information according to the double-flow heterogeneous deep network architecture, and joint distribution of visual-text features of the industrial equipment is learned to obtain defect feature vectors of the industrial equipment; performing multi-dimensional matching on the defect feature vectors and similar cases in a maintenance knowledge base, and screening treatment schemes corresponding to the defect feature vectors to obtain a maintenance strategy; and issuing the defect information corresponding to the defect feature vector and the maintenance strategy to a communication terminal of a maintainer. According to the method and the device, the problem of low cross-modal information utilization rate caused by multi-modal data splitting and no feature level fusion in the prior art can be solved.

Description

Technical Field

[0001] This application belongs to the field of intelligent manufacturing technology, and particularly relates to a method, device, equipment and medium for autonomous decision-making of defects for industrial inspection tours. Background Art

[0002] With the in-depth development of intelligent manufacturing from automation to intelligence, the predictive maintenance of industrial equipment has become the core link of intelligent manufacturing. The predictive maintenance of industrial equipment mainly relies on industrial inspection tours. Currently, industrial inspection tours mainly adopt a collaborative mode of manual inspection tours and sensor networks (including temperature and humidity / smoke sensors and monitoring equipment, etc.). This mode includes two implementation paths: First, the device images are collected on-site and sent to maintenance personnel, and manual experience is relied on to judge the type of defects and formulate disposal plans. Second, the abnormal areas of the device are detected through an image model, and maintenance personnel with repair experience are notified to carry out repairs. The above methods have the following problems: (1) Severe personnel dependence: Experienced maintenance personnel are required to participate in the entire decision-making process, resulting in high labor costs, subjective judgment errors, and a lag in response timeliness. (2) Breakage of the decision-making chain: There is a lack of intelligent association between the defect recognition results and the maintenance knowledge base, and the reuse rate of historical cases is low, forming an "inspection - decision-making" information island.

[0003] The core reason for the above problems lies in the insufficient fusion of multi-modal data. Specifically: First, the defect recognition and disposal strategy are disjointed: In the existing technology, the image recognition model usually operates independently of the maintenance knowledge base, resulting in the disconnection between the defect recognition results and the generation link of the disposal strategy. For example, although the anomaly detection system based on a large vision model can locate the surface defects of the device, it lacks the ability to mine the causal associations hidden in historical maintenance cases (such as the mapping relationship between specific defect types and the material aging cycle), and additional manual experience is required to complete the strategy matching. Additionally, the data is prone to contamination: At the data level, the detection results of a single-modal sensor (such as infrared thermal imaging) are easily affected by the environment (such as device surface reflection, steam occlusion), and multi-modal data only adopts a simple weighted fusion method without establishing a cross-modal feature alignment mechanism (such as the spatio-temporal correlation between hot spot distribution and vibration spectrum), resulting in an increase in the model's false detection rate.

[0004] The above defects are mainly due to the fragmentation of multi-modal data and the lack of feature-level fusion, resulting in low utilization rate of cross-modal information, making it difficult for existing solutions to meet the requirements of minute-level response and high-precision decision-making in complex industrial scenarios. Summary of the Invention

[0005] This application provides a defect autonomous decision-making solution for industrial inspection, realizing device defect autonomous decision-making based on the "edge-cloud-library" collaborative architecture, constructing a multi-modal feature fusion engine and a dynamic knowledge evolution system, capable of fusing dynamic knowledge evolution and multi-modal deep reasoning for autonomous decision-making. Specifically, by fusing multi-spectral imaging data such as visible light, infrared, and ultraviolet with device operation status parameters, combined with remote large model reasoning and dynamic knowledge graph update mechanism, it realizes the full-process closed-loop autonomy of device defect detection, maintenance strategy matching, and disposal effect evaluation. Through the above method, it can perform feature-level fusion on multi-modal data, avoid the fragmentation of multi-modal data, improve the utilization rate of cross-modal information, and provide a real-time and accurate decision support system for predictive maintenance of industrial equipment.

[0006] According to the first aspect of this application, an embodiment of this application provides a defect autonomous decision-making method for industrial inspection, including: Use a multi-spectral fusion terminal to collect the device operation status information of industrial equipment and upload it to the cloud server; Use the cross-modal depth analysis model built in the cloud server to perform cross-modal learning on the device operation status information according to the dual-stream heterogeneous deep network architecture, learn the joint distribution of the visual-text features of the industrial equipment, and obtain the defect feature vector of the industrial equipment; According to the hybrid dual-engine reasoning mode, perform multi-dimensional matching and screening on the defect feature vector and similar cases in the maintenance knowledge base to obtain the disposal plan corresponding to the defect feature vector, and obtain the maintenance strategy; Send the defect information and maintenance strategy corresponding to the defect feature vector to the communication terminal of the maintenance personnel.

[0007] Preferably, after the step of sending the defect information and maintenance strategy corresponding to the defect feature vector to the communication terminal of the maintenance personnel in the above device defect autonomous decision-making method, it further includes: Obtain the maintenance scoring signal of the industrial equipment; Adopt knowledge distillation technology, use the maintenance scoring signal to adjust the node weights of the corresponding dynamic knowledge graph in the maintenance knowledge base, and establish a mapping relationship between the score and the weight; Use the mapping relationship between the score and the weight to optimize the maintenance knowledge base.

[0008] Preferably, in the above device defect autonomous decision-making method, the step of using a multi-spectral fusion terminal to collect the device operation status information of industrial equipment and upload it to the cloud server includes: Use a multi-spectral fusion terminal to collect the multi-spectral status image of industrial equipment; Use the 3D contour calibration algorithm to compare the spatial coordinates of the industrial equipment in the multi-spectral status image with the corresponding spatial features in the preset CAD model in real time; Perform spatial coordinate calibration on the spatial characteristics of industrial equipment in combination with the gyroscope attitude compensation mechanism; Perform filtering and denoising on the multi-spectral status image after spatial coordinate calibration to obtain the equipment operation status information of the industrial equipment.

[0009] Preferably, in the above-mentioned equipment defect autonomous decision-making method, the step of using the cross-modal depth analysis model built in the cloud server to perform cross-modal learning on the equipment operation status information according to the dual-stream heterogeneous depth network architecture, learning the joint distribution of the visual-text features of the industrial equipment, and obtaining the defect feature vector of the industrial equipment includes: Upload the equipment operation status information of the industrial equipment to the cross-modal depth analysis model of the cloud server. Among them, there is a dual-stream heterogeneous depth network architecture in the cross-modal depth analysis module, and the equipment operation status information includes multi-spectral status images and equipment status data; Control the visual processing stream of the dual-stream heterogeneous depth network architecture, use the object detection algorithm combined with the multi-scale attention mechanism to establish feature associations of the multi-spectral status image in multiple scale dimensions, and detect the visual features of the multi-spectral status image; Control the text processing stream of the dual-stream heterogeneous depth network architecture, use the large text processing model to construct a domain semantic parsing dictionary, and use the domain semantic parsing dictionary to parse the text features of the equipment status image; Use the cross-modal adversarial distillation pipeline to adversarially collect visual features and text features; Use the generator network built by the cross-modal adversarial distillation pipeline to learn the joint distribution of visual features and text features according to the alignment loss function, and obtain the defect feature vector.

[0010] Preferably, in the above-mentioned equipment defect autonomous decision-making method, the step of controlling the visual processing stream of the dual-stream heterogeneous depth network architecture, using the object detection algorithm combined with the multi-scale attention mechanism to establish feature associations of the multi-spectral status image in multiple scale dimensions, and detecting the visual features of the multi-spectral status image includes: Control the visual processing stream to adopt the improved ViT-Transformer algorithm, introduce a deformable convolutional layer to dynamically perceive the convolutional kernel offset of the multi-spectral status image, and capture the equipment defects in the multi-spectral status image; Introduce a multi-scale attention mechanism in the visual processing stream to establish feature associations of the multi-spectral status image in multiple scale dimensions, and identify the fuzzy defects of the industrial equipment; Obtain the visual features corresponding to the image defects and fuzzy defects.

[0011] Preferably, in the above-mentioned method for autonomous decision-making of equipment defects, according to the hybrid dual-engine reasoning mode, the step of performing multi-dimensional matching on the defect feature vector and similar cases in the maintenance knowledge base and screening the disposal plan corresponding to the defect feature vector to obtain a maintenance strategy includes: Call the dynamic knowledge graph corresponding to the maintenance knowledge base, perform multi-dimensional matching on the defect feature vector and similar cases, and calculate the cosine similarity between the defect feature and the similar cases according to the multi-dimensional matching degree; Extract a predetermined number of maintenance cases with a cosine similarity above the similarity threshold; And, Construct an expert experience decision tree based on fuzzy logic, and use the expert experience decision tree to process the uncertain conditions corresponding to the maintenance cases through a fuzzy membership function to obtain a disposal plan; Integrate the maintenance cases and disposal plans to obtain a maintenance strategy.

[0012] Preferably, in the above-mentioned method for autonomous decision-making of equipment defects, after the step of sending the defect information and maintenance strategy corresponding to the defect feature vector to the communication terminal of the maintenance personnel, it further includes: After the industrial equipment is repaired, continuously collect the operating parameters of the industrial equipment; Input the operating parameters into the health index model to obtain the improvement rate of the industrial equipment after repair; Use the improvement rate to evaluate the maintenance effect of the maintenance personnel; When the improvement rate is less than or equal to a predetermined improvement threshold, trigger the parameter update mechanism of the cross-modal depth analysis model; Obtain the maintenance scoring signal uploaded by the communication terminal; Use the maintenance scoring signal to iteratively update the node weights of the dynamic knowledge graph corresponding to the maintenance knowledge base through knowledge distillation technology.

[0013] According to the second aspect of the present application, the present application also provides a defect autonomous decision-making device for industrial inspection, including: An information collection module, configured to use a multi-spectral fusion terminal to collect the equipment operation status information of industrial equipment and upload it to a cloud server; A cross-modal learning module, configured to use the cross-modal depth analysis model built in the cloud server to perform cross-modal learning on the equipment operation status information according to the dual-stream heterogeneous deep network architecture, learn the joint distribution of the visual-text features of the industrial equipment, and obtain the defect feature vector of the industrial equipment; A strategy learning module, configured to perform multi-dimensional matching on the defect feature vector and similar cases in the maintenance knowledge base according to the hybrid dual-engine reasoning mode and screen the disposal plan corresponding to the defect feature vector to obtain a maintenance strategy; An information distribution module, configured to distribute the defect information corresponding to the defect feature vector and the maintenance strategy to the communication terminal of the maintenance personnel.

[0014] Preferably, the above-mentioned device defect autonomous decision-making device further includes: a scoring optimization module, configured to obtain a maintenance scoring signal of the industrial device; adopt a knowledge distillation technique to use the maintenance scoring signal to adjust the node weights of the corresponding dynamic knowledge graph in the maintenance knowledge base, and establish a mapping relationship between the score and the weight; use the mapping relationship between the score and the weight to optimize the maintenance knowledge base.

[0015] Preferably, in the above-mentioned device defect autonomous decision-making device, the information acquisition module is specifically configured to use a multi-spectral fusion terminal to collect multi-spectral status images of the industrial device; use a 3D contour calibration algorithm to compare the spatial coordinates of the industrial device in the multi-spectral status image with the corresponding spatial features in the preset CAD model in real time; combine the gyroscope attitude compensation mechanism to perform spatial coordinate calibration on the spatial features of the industrial device; perform filtering and denoising processing on the multi-spectral status image after spatial coordinate calibration to obtain the device operation status information of the industrial device.

[0016] Preferably, in the above-mentioned device defect autonomous decision-making device, the cross-modal learning module is specifically configured to upload the device operation status information of the industrial device to the cross-modal depth analysis model of the cloud server, where the cross-modal depth analysis module has a dual-stream heterogeneous deep network architecture, and the device operation status information includes multi-spectral status images and device status data; control the visual processing stream of the dual-stream heterogeneous deep network architecture, use an object detection algorithm combined with a multi-scale attention mechanism to establish feature associations of the multi-spectral status image in multiple scale dimensions, and detect the visual features of the multi-spectral status image; control the text processing stream of the dual-stream heterogeneous deep network architecture, use a large text processing model to construct a domain semantic parsing dictionary, and use the domain semantic parsing dictionary to parse the text features corresponding to the device status image; use a cross-modal adversarial distillation pipeline to adversarially collect visual features and text features; use the generator network constructed by the cross-modal adversarial distillation pipeline to learn the joint distribution of visual features and text features according to the alignment loss function to obtain a defect feature vector.

[0017] Preferably, the above-mentioned cross-modal learning module is specifically configured to control the visual processing stream to adopt an improved ViT-Transformer algorithm, introduce a deformable convolutional layer to dynamically perceive the convolutional kernel offset of the multi-spectral status image, and capture the device defects in the multi-spectral status image; introduce a multi-scale attention mechanism in the visual processing stream to establish feature associations of the multi-spectral status image in multiple scale dimensions, and identify the fuzzy defects of the industrial device; obtain the visual features corresponding to the image defects and the fuzzy defects.

[0018] Preferably, the above-mentioned policy learning module is specifically configured to call the dynamic knowledge graph corresponding to the maintenance knowledge base, perform multi-dimensional matching on the defect feature vector and similar cases, and calculate the cosine similarity between the defect feature and the similar cases according to the multi-dimensional matching degree; extract a predetermined number of maintenance cases with the cosine similarity above the similarity threshold; and, construct an expert experience decision tree based on fuzzy logic, use the expert experience decision tree to process the uncertain conditions corresponding to the maintenance cases through a fuzzy membership function to obtain a disposal plan; and synthesize the maintenance cases and the disposal plan to obtain a maintenance strategy.

[0019] Preferably, the above-mentioned device further includes: a negative feedback adjustment module, configured to continuously collect the operating parameters of the industrial equipment after the industrial equipment is repaired; input the operating parameters into the health index model to obtain the improvement rate of the industrial equipment after repair; use the improvement rate to evaluate the repair and disposal effect of the maintenance personnel; when the improvement rate is less than or equal to a predetermined improvement threshold, trigger the parameter update mechanism of the cross-modal depth analysis model.

[0020] According to the third aspect of the present application, the present application further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the defect autonomous decision-making method for industrial inspection provided by any of the above technical solutions.

[0021] According to the fourth aspect of the present application, the present application further provides a computer storage medium, on which computer executable instructions are stored. When the computer program is executed by the processor, it implements the defect autonomous decision-making method for industrial inspection provided by any of the above technical solutions.

[0022] The technical solution of the present application at least has the following technical effects: The technical solution for autonomous defect decision-making for industrial inspection provided by this application can collect the device operation status information of various industrial devices in multiple modalities through multiple spectral image signals by using a multi-spectral fusion terminal. Then, a cross-modal depth analysis model built in the cloud server is used to establish the feature and channel fusion of the above-mentioned device operation status information in multiple modalities. Specifically, a dual-stream heterogeneous deep network architecture is used to perform cross-modal learning on the above-mentioned device operation status information, learn the joint distribution of the visual-text features of the industrial device, and then obtain the defect feature vector of the industrial device, thereby realizing the utilization of cross-modal and cross-dimensional information. Then, according to the hybrid dual-engine inference mode, multi-dimensional matching is performed between the defect feature vector and similar cases in the maintenance knowledge base, and the disposal plan is screened. Combining the matched cases and disposal plans can automatically obtain the maintenance strategy for the defect features of the industrial device, and the decision-making process does not need to rely on technical personnel. Finally, the defect information and maintenance strategy corresponding to the defect feature vector are sent to the communication terminal of the maintenance personnel, and the maintenance of industrial device defects can be realized. Through the above solution, the problem in the prior art that multi-modal data is fragmented and feature-level fusion cannot be effectively established, resulting in low cross-modal information utilization rate, can be solved. Furthermore, the full-process closed-loop automation of device defect detection, autonomous decision-making of maintenance strategies, and device maintenance disposal can be realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of this application, and constitute a part of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation of this application. In the drawings: Figure 1 FIG. is a schematic diagram of an application scenario provided by an embodiment of this application; Figure 2 FIG. is a flowchart of a first method for autonomous defect decision-making for industrial inspection provided by an embodiment of this application; Figure 3 is Figure 1 a flowchart of a method for obtaining device operation status information provided by the embodiment shown; Figure 4 is Figure 1 a flowchart of a cross-modal learning method for device operation status information provided by the embodiment shown; Figure 5 is Figure 4 a flowchart of a method for detecting visual features provided by the embodiment shown; Figure 6 is Figure 1 a flowchart of a method for matching maintenance strategies provided by the embodiment shown; Figure 7 is Figure 1Flow chart of a method for issuing defect information and maintenance strategies provided by the illustrated embodiment; Figure 8 Flow chart of a method for regulating a maintenance knowledge base using a scoring signal negative feedback provided by an embodiment of the present application; Figure 9 Flow chart of a second defect autonomous decision-making method for industrial inspection provided by an embodiment of the present application; Figure 10 Structural diagram of a defect autonomous decision-making device for industrial inspection provided by an embodiment of the present application; Figure 11 Structural diagram of an electronic device provided by an embodiment of the present application; Figure 12 Overall architecture diagram of a defect autonomous decision-making for industrial inspection provided by an embodiment of the present application; Figure 13 Architecture diagram of a cross-modal adversarial distillation pipeline provided by an embodiment of the present application; Figure 14 Architecture diagram of a maintenance scoring signal combined with knowledge distillation technology provided by an embodiment of the present application. Detailed implementation manners

[0024] To more clearly illustrate the overall concept of the present application, the following will be described in detail by way of examples in conjunction with the accompanying drawings of the specification.

[0025] In the following description, many specific details are set forth in order to fully understand the present application. However, the present application may be implemented in other ways different from those described herein. Therefore, the protection scope of the present application is not limited by the specific embodiments disclosed below. It should be noted that, without conflict, the embodiments of the present application and the features in each embodiment may be combined with each other.

[0026] In the present application, unless otherwise clearly defined and limited, the first feature may be in direct contact with the second feature "above" or "below", or the first and second features may be indirectly in contact through an intermediate medium. In the description of this specification, the description of reference terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0027] The prior art has the following defects: I. Severe personnel dependence: Experienced maintenance personnel are required to participate in the entire decision-making process, resulting in high labor costs, subjective judgment biases, and delayed response timeliness. II. Broken decision-making chain: There is a lack of intelligent association between defect identification results and the maintenance knowledge base, and the reuse rate of historical cases is low, forming an "inspection - decision-making" information island.

[0028] The above defects stem from threefold fragmentation of the traditional technical architecture: (1) Feature-level channel fusion of multi-modal data (images / sensor data / maintenance records) has not been established, resulting in low utilization of cross-dimensional information; (2) The decision-making process overly relies on experienced technical personnel; (3) The update of the knowledge base depends on manual input, lacking a self-evolution mechanism based on on-site feedback.

[0029] To solve the above technical problems, see Figure 1 , the following embodiments of this application provide a defect autonomous decision-making method for industrial inspection, including a handheld multi-spectral fusion terminal 1, a cloud server 2, and a communication terminal 4 of an operator; among them, the cloud server 2 includes a maintenance knowledge base 3; it can integrate terminal acquisition, cloud retrieval, knowledge base matching, and knowledge recommendation, thus effectively solving the problems existing in the above methods. Specifically, by fusing the visible light, infrared, and ultraviolet multi-spectral imaging data of the multi-spectral fusion terminal 1 with the operating state parameters of the industrial equipment 5, combined with the large model reasoning and dynamic knowledge graph update mechanism of the cloud server 2, a maintenance strategy is formed and sent to the communication terminal 4 of the maintenance personnel. Through the above method, a full-process closed-loop autonomy of equipment defect detection, maintenance strategy matching, and disposal effect evaluation can be achieved. This application can mainly solve the defects such as fragmented processing of multi-modal data and reliance on manual experience in traditional inspections, resulting in response delays and insufficient standardization of disposal plans, and provide a real-time and accurate decision support system for predictive maintenance of industrial equipment.

[0030] Specifically, to achieve the above purpose, see Figure 2 , Figure 2 is a schematic flow diagram of a defect autonomous decision-making method for industrial inspection provided by an embodiment of this application. As Figure 2 shown, the defect autonomous decision-making method for industrial inspection includes: S110: Use the multi-spectral fusion terminal to collect the equipment operating state information of the industrial equipment and upload it to the cloud server.

[0031] Combined with Figure 1As can be seen from the application scenario diagram shown, the inspection personnel use a handheld multi-spectral fusion terminal (which can integrate visible light, infrared, and ultraviolet three-channel imaging modules) to collect multi-spectral status images of industrial equipment, and the multi-spectral status images can reflect the operating status of the industrial equipment. The multi-spectral fusion terminal is built-in with a basic anti-shake algorithm based on gyroscope attitude compensation and a device contour matching function. The device contour matching function can select the Scale-Invariant Feature Transform (SIFT) feature point detection algorithm to realize rotation correction and median filtering denoising of the collected images. The preprocessed multi-spectral status images will be uploaded to the cloud after being bound with the device ID and the acquisition timestamp. The multi-spectral fusion terminal collects multi-spectral status images, and can obtain multi-modal operating status data of industrial equipment.

[0032] Specifically, as a preferred embodiment, as Figure 3 shown, the step S110: using the multi-spectral fusion terminal to collect the operating status information of the industrial equipment and upload it to the cloud server includes: S111: Using the multi-spectral fusion terminal to collect multi-spectral status images of the industrial equipment; S112: Using a 3D contour calibration algorithm to compare the spatial coordinates of the industrial equipment in the multi-spectral status image with the corresponding spatial features in the preset CAD model in real time; S113: Combining the gyroscope attitude compensation mechanism to calibrate the spatial coordinates of the spatial features of the industrial equipment; S114: Performing filtering and denoising processing on the multi-spectral status image after spatial coordinate calibration to obtain the operating status information of the industrial equipment.

[0033] The technical solution provided by the embodiment of the present application can realize multi-modal image data acquisition. In a specific embodiment, the inspection personnel use a dedicated three-spectral fusion handheld terminal (integrating visible light, infrared, and ultraviolet three-channel imaging modules) to collect equipment status data. When a specific defect mode is selected (such as local overheating), the multi-spectral fusion terminal automatically switches to the optimal imaging combination (such as infrared main channel + ultraviolet auxiliary channel) based on a preset strategy. To solve the problem of imaging distortion under complex working conditions, a 3D contour calibration algorithm based on the device CAD model is introduced: by comparing the spatial features of the collected image with the preset CAD model in real time, and combining the gyroscope attitude compensation module for spatial coordinate calibration, the distortion rate of imaging is reduced. The collected data is processed in the spatial domain to eliminate the shooting angle deviation and denoised, which will effectively improve the availability of the original data in various environments.

[0034] The prior art has three structural defects: First, multi-modal perception is fragmented; second, the decision-making process is black-boxed; third, the decision-making process is black-boxed. Regarding the fragmentation of multi-modal perception, the visible light, infrared, and ultraviolet imaging data collected by traditional technologies are analyzed independently from the sensor parameters, lacking a cross-modal feature correlation model. To solve this problem, this application builds a cross-modal depth analysis model in the cloud to achieve cross-modal fusion of various imaging data and sensor parameters. Specifically: Figure 2 For the technical solution provided by the illustrated embodiment, after step S110: using a multi-spectral fusion terminal to collect the device operation status information of an industrial device, the following steps are further included: S120: Using the cross-modal depth analysis model built in the cloud server, perform cross-modal learning on the device operation status information according to the dual-stream heterogeneous depth network architecture, learn the joint distribution of the visual-text features of the industrial device, and obtain the defect feature vector of the industrial device.

[0035] In the embodiment of this application, the extraction and classification of defect features can be achieved through the cross-modal depth analysis model. Specifically, an improved YOLOv5 detection model is deployed on the cloud server. The improvement of this improved YOLOv5 detection model lies in introducing an attention mechanism in its classification layer. Specifically, multi-scale defect features can be extracted through the Feature Pyramid Network (FPN). An attention mechanism (SE module) is introduced into the detection model. The attention mechanism SE module can play the roles of calibration, attention, and aggregation, thereby improving the recognition accuracy of tiny defects and extracting the defect features that are focused on. The output result of the detected defect features includes the defect location, type, and confidence index. Specifically, the spatial attention mechanism SE adds attention in the channel dimension. After placing the SE module behind the Spatial Pyramid Pooling Fast (SPPF), the network is recalibrated.

[0036] Specifically, as a preferred embodiment, as Figure 4 shown, this step S120: using the cross-modal depth analysis model built in the cloud server, perform cross-modal learning on the device operation status information according to the dual-stream heterogeneous depth network architecture, learn the joint distribution of the visual-text features of the industrial device, and obtain the defect feature vector of the industrial device, includes: S121: Upload the device operation status information of industrial equipment to the cross-modal depth analysis model on the cloud server. Among them, the cross-modal depth analysis module has a dual-stream heterogeneous depth network architecture, and the device operation status information includes multi-spectral status images and device status data. The large model deployed on the cloud, that is, the cross-modal depth analysis model, its dual-stream heterogeneous depth network architecture includes a visual processing stream (i.e., image encoder) and a text processing stream (i.e., text encoder); it can respectively learn the visual features and text features related to the above device operation status, and finally combine the above visual features and text features to obtain a defect vector.

[0037] S122: Control the visual processing stream of the dual-stream heterogeneous depth network architecture, use the object detection algorithm combined with the multi-scale attention mechanism to establish the feature correlation of the multi-spectral status image in multiple scale dimensions, and detect the visual features of the multi-spectral status image. The object detection algorithm provided in the embodiments of the present application can select the improved YOLOv5 detection model, and the improvement of the detection model lies in that an attention mechanism (SE module) is introduced into its classification layer, and multi-scale defect features are extracted through the Feature Pyramid Network (FPN).

[0038] S123: Control the text processing stream of the dual-stream heterogeneous depth network architecture, use the text processing large model to construct a domain semantic parsing dictionary, and use the domain semantic parsing dictionary to parse the text features of the device status image. The text processing large model can be a GPT-3 model based on knowledge enhancement. By injecting the structured knowledge of device maintenance records (such as the association rule of "bearing abnormal noise - insufficient lubrication - replace grease"), a domain-specific semantic parsing dictionary is constructed, so that the text features corresponding to the device status image can be parsed using this dictionary.

[0039] S124: Use the cross-modal adversarial distillation pipeline to adversarially collect visual features and text features. For example: The cross-modal adversarial distillation pipeline has the same dimension as the visual features and text features. In this way, the features of the two modalities adversarially compete for the dimension of this distillation pipeline, and the above visual features and text features can be adversarially collected, so as to realize the cross-modal collection and fusion of the features of the two modalities, or even multiple modalities.

[0040] S125: Use the generator network constructed by the cross-modal adversarial distillation pipeline to learn the joint distribution of visual features and text features according to the alignment loss function, and obtain the defect feature vector. See Figure 13 , the cross-modal adversarial distillation pipeline includes a generator network, a discriminator network, and a feature alignment loss function. The generator network mainly includes two functions: learning the joint distribution of cross-modal features and generating fused features.

[0041] The generator network is the core execution engine of the cross-modal adversarial distillation pipeline. It can learn the joint distribution of the two modal features through cross-modal acquisition and adversarial learning of visual features and text features, thereby solving the problem of multi-modal data fragmentation and inability to perform feature-level fusion in the prior art. Specifically, both the visual feature and the text feature are 128-dimensional, so that the generator network can compress the 256-dimensional concatenated vector (128 + 128) to 128 dimensions.

[0042] As a specific embodiment, the present application uploads device image data to a cloud large model, and the cloud large model can detect defect information corresponding to the device image data. The cloud large model is designed with a dual-stream heterogeneous deep network architecture, and the dual-stream heterogeneous deep network architecture includes a visual processing stream and a text processing stream. Among them, combined with Figure 12 the shown architecture, the visual processing stream (i.e., the image encoder) uses an improved ViT-Transformer model (input size 384×384) as the model of the object detection algorithm. The improvement of this model is to introduce a deformable convolutional layer (Deformable DCNN) into the standard module of the original ViT-Transformer model. DCNN can dynamically perceive the offset of the convolutional kernel (offset range ±5 pixels), and effectively capture tiny defects in the multi-spectral image (such as cracks with a size <0.5mm 2 . At the same time, a multi-scale attention mechanism is introduced to establish feature associations in the dimensions of 16×16, 32×32, and 64×64, so as to improve the recognition ability of fuzzy defects. Through this visual processing stream, a visual feature vector can be obtained. The text processing stream (i.e., the text encoder) is based on the knowledge-enhanced GPT-3 model. By injecting structured knowledge of device maintenance records (such as the association rule of "bearing abnormal noise - insufficient lubrication - replace grease"), a domain-specific semantic parsing dictionary is constructed to obtain a text feature vector. The two modal features (visual feature and text feature) generate 128-dimensional semantic vectors after independent encoding. A cross-modal adversarial distillation pipeline is designed, and a generator network (including 4 fully connected layers) is constructed in the cross-modal adversarial distillation pipeline to learn the joint distribution of visual-text features, and finally a defect vector is obtained. This defect feature vector is then matched with the knowledge base strategy. The adversarial distillation pipeline is 128-dimensional. In this way, for two modal features with a dimension higher than or equal to 128, there is a competitive relationship when input into the adversarial distillation pipeline, and they need to compete for the dimensions of this pipeline, so as to realize the joint distribution of visual-text features. In this way, the generator network can learn the joint distribution of this visual-text feature according to the loss function and obtain the defect feature vector. Among them, the alignment loss function is defined as:

[0043] Among them, is the visual encoder (Image Encoder) for the one image sample the extracted feature vector is the feature vector extracted by the Text Encoder for the th text sample extracted

[0044] As a preferred embodiment, for the above step S122: controlling the visual processing stream of the dual-stream heterogeneous deep network architecture, using an object detection algorithm combined with a multi-scale attention mechanism to establish feature associations of the multi-spectral state image in multiple scale dimensions, and detecting the visual features of the multi-spectral state image. Specifically, refer to Figure 5 : S1221: Control the visual processing stream to adopt an improved ViT-Transformer algorithm, introduce a deformable convolutional layer to dynamically perceive the convolutional kernel offset of the multi-spectral state image, and capture the device defects in the multi-spectral state image; S1222: Introduce a multi-scale attention mechanism in the visual processing stream to establish feature associations of the multi-spectral state image in multiple scale dimensions, and identify the fuzzy defects of industrial equipment; S1223: Obtain the visual features corresponding to the image defects and fuzzy defects.

[0045] For the technical solution provided by the embodiment of the present application, the selected object detection algorithm can be an improved ViT-Transformer model (input size 384×384), and a deformable convolutional layer (Deformable DCNN) is introduced into the standard module of the model. The DCNN is used to dynamically perceive the convolutional kernel offset (offset range ±5 pixels), and can effectively capture the tiny defects in the multi-spectral image (such as cracks with a size <0.5mm 2 ); at the same time, a multi-scale attention mechanism is introduced to establish feature associations in the dimensions of 16×16, 32×32, and 64×64, and the recognition ability for fuzzy defects is improved.

[0046] Among the three major defects of the prior art, for the black box of the decision-making process, that is, although the deep learning model can output the defect classification result, it lacks an interpretable maintenance strategy derivation link and a knowledge matching process. In the following embodiments of the present application, by comparing the similarity between the defect feature vector and the similar cases in the maintenance knowledge base, the closest maintenance case is matched, so as to generate a disposal plan for the device defect.

[0047] Specifically, Figure 2For the technical solution provided by the illustrated embodiment, after the step of using the cross-modal depth analysis model built in the cloud server to perform cross-modal learning on the device operation status information according to the dual-stream heterogeneous depth network architecture, and learning the joint distribution of the visual-text features of the industrial device to obtain the defect feature vector of the industrial device, the method further includes: S130: According to the hybrid dual-engine inference mode, perform multi-dimensional matching on the defect feature vector and similar cases in the maintenance knowledge base, and screen the disposal solutions corresponding to the defect feature vector to obtain a maintenance strategy.

[0048] In the embodiment of the present application, by retrieving the local maintenance knowledge base, which stores standard cases such as maintenance work orders and maintenance records generated during daily maintenance, and then calculating the cosine similarity between the current 128-dimensional defect feature vector and the historical cases in the maintenance knowledge base. Specifically, if the cosine similarity is greater than or equal to a certain threshold (threshold > 0.75), return the Top5 candidate cases; if the number of cases does not meet 5, no return is made. Screen the disposal solutions that meet the current working conditions (such as "give priority to shutdown inspection when the temperature > 80°C") through a rule engine based on a decision tree. It should be noted that the defect feature vector belongs to the features fused later, and 128 dimensions represent the dimension of this defect feature vector.

[0049] As a preferred embodiment, as Figure 6 shown, in the above device defect autonomous decision-making method, step S130: According to the hybrid dual-engine inference mode, perform multi-dimensional matching on the defect feature vector and similar cases in the maintenance knowledge base, and screen the disposal solutions corresponding to the defect feature vector to obtain a maintenance strategy, specifically including: S131: Invoke the dynamic knowledge graph corresponding to the maintenance knowledge base, perform multi-dimensional matching on the defect feature vector and similar cases, and calculate the cosine similarity between the defect feature and the similar cases according to the multi-dimensional matching degree. The calculation formula of this cosine similarity is as follows: Suppose the n-dimensional defect feature vectors A=(a1,a2,…a n ) and B=(b1,b2,…,b n ), in the embodiment of the present application, usually n = 128, then the calculation formula of the cosine similarity is as follows Consine Similarity(A,B) =

[0050] Among them, the numerator part represents the inner product of vectors A and B, and the denominator part represents the product of the moduli of the two vectors.

[0051] S132: Extract a predetermined number of maintenance cases with a cosine similarity above the similarity threshold.

[0052] And, S133: Construct an expert experience decision tree based on fuzzy logic. Use the expert experience decision tree to process the uncertain conditions corresponding to the maintenance cases through the fuzzy membership function to obtain the disposal plan. From the rules in the knowledge base, real-time working conditions, etc. The defect feature vector (128-dimensional) retrieves historical cases through cosine similarity, and the expert experience decision tree based on fuzzy logic introduces real-time working condition constraints on this basis to achieve dynamic correction.

[0053] S134: Synthesize the maintenance cases and the disposal plan to obtain the maintenance strategy.

[0054] The technical solution provided by the embodiment of the present application retrieves similar cases through the maintenance knowledge base, matches the maintenance strategy, and generates the disposal plan. Specifically, it includes: calling the dynamic knowledge graph corresponding to the maintenance knowledge base, performing multi-dimensional matching on the above defect feature vector, calculating the cosine similarity between the current defect feature vector and the historical cases, and screening out the candidate cases. The dimensions that need to be matched here include equipment, environment, resources, etc.

[0055] The reasoning process of the maintenance strategy in the embodiment of the present application includes two parts. Specifically, a hybrid dual-engine reasoning mode is adopted, with a case reasoning module and a rule reasoning module. Among them, the case-based reasoning (CBR) module is used to calculate the cosine similarity between the current defect feature vector and the historical cases (threshold > 0.85), screen out the top 10 cases as candidate cases and extract the high-frequency disposal items (such as the lubricant replacement ratio is 82% in the case of "bearing overheating"); the rule-based reasoning (RBR) module constructs an expert experience decision tree based on fuzzy logic, and uses the fuzzy membership function to process uncertain conditions (such as "temperature is on the high side" is defined as a Gaussian distribution in the interval of 70 - 90 °C), to achieve human-like decision-making. The knowledge distillation technology is used to convert the maintenance scoring signal uploaded by the operator into the adjustment amount of the graph node weight, establish the mapping relationship between the score and the weight, and realize the progressive optimization of the knowledge base.

[0056] Figure 2 The technical solution provided by the shown embodiment, after obtaining the maintenance strategy, further includes the following steps: S140: Send the defect information corresponding to the defect feature vector and the maintenance strategy to the communication terminal of the maintenance personnel.

[0057] After the maintenance personnel receive the maintenance decision and perform the maintenance, they will continuously collect the equipment operation parameters (temperature, vibration value, etc.) to calculate the equipment health index (HI). When the improvement rate of HI is less than or equal to a certain degree, it triggers the analysis of the maintenance cases in the maintenance knowledge base, manually corrects the maintenance decision, and regularly fine-tunes the parameters of the above cross-modal depth analysis model (such as the parameters related to the convolution kernel).

[0058] Specifically, as a preferred embodiment, such as Figure 7As shown, after step S140: the step of sending the defect information and maintenance strategy corresponding to the defect feature vector to the communication terminal of the maintenance personnel, the following steps are further included: S141: After the industrial equipment is repaired, continuously collect the operating parameters of the industrial equipment; S142: Input the operating parameters into the health index model to obtain the improvement rate of the industrial equipment after repair; S143: Use the improvement rate to evaluate the maintenance disposal effect of the maintenance personnel; S144: When the improvement rate is less than or equal to the predetermined improvement threshold, trigger the parameter update mechanism of the cross-modal depth analysis model; S145: Obtain the maintenance scoring signal uploaded by the communication terminal; the maintenance scoring signal here is generated in combination with the improvement rate.

[0059] S146: Use the maintenance scoring signal to iteratively update the node weights of the corresponding dynamic knowledge graph of the maintenance knowledge base through knowledge distillation technology.

[0060] Knowledge distillation is a model compression and acceleration technology that aims to transfer the knowledge learned by a large model (usually called the teacher model) to a small model (usually called the student model), so that the small model can achieve performance close to that of the large model while reducing computational resource consumption and inference time.

[0061] As Figure 14 shown, in the embodiment of the present application, the subjective scoring (maintenance scoring signal) of the maintenance personnel for the maintenance plan is characterized as the teacher signal, representing the actual evaluation result of the plan; the decision suggestion currently output by the dynamic knowledge graph is regarded as the student signal, reflecting the inference result of the model. In the embodiment of the present application, a mapping is constructed through knowledge distillation technology. When the maintenance scoring signal is received, the system locates the graph nodes of the dynamic knowledge graph directly associated with the plan, calculates the weight change value through the mapping function, and the level of the score affects the adjustment direction. By comparing the differences between the teacher signal and the student signal, the weight distribution of the associated nodes in the knowledge graph is dynamically adjusted - this process improves the priority of high-scoring plans in subsequent case matching and suppresses low-scoring plans, realizing the progressive optimization of the cloud AI model (the above cross-modal depth analysis model).

[0062] In the solution provided by the embodiment of the present application, the defect information and maintenance strategy are sent to the maintenance personnel through cloud messages. After the maintenance personnel perform the maintenance operation, they continuously collect the equipment operating parameters (vibration value, temperature curve, etc.), and evaluate the disposal effect through the health index model. Taking the defect information detected by the industrial equipment: temperature, vibration value, and current output power as an example, the health index models corresponding to the three indicators are as follows:

[0063] When applied to the scenario of detecting whether the surface temperature of the device is abnormal, in the above formula represents the temperature change of the key part of the device, that is, the absolute value of the temperature difference at the same monitoring point before and after maintenance; represents the standard temperature value of the device under normal working conditions; is the effective value of the current vibration amplitude of the device, which characterizes the stability of the mechanical structure. The vibration signal can be collected by an acceleration sensor (such as a piezoelectric sensor); represents the reference vibration amplitude of the device in a healthy state; represents the actual measured value of the current output power of the device; is the rated power of the device; the above weights (0.5, 0.3, and 0.2) respectively reflect the contribution weights of temperature, vibration, and power to the health state of the device, and different devices can be replaced with other evaluation indicators.

[0064] In the embodiment of the present application, when the HI improvement rate < 30%, a necessary model update is triggered. The daily maintenance score (1 - 5 stars) updates the weights of the graph nodes through knowledge distillation, and forms a closed-loop feedback system to iteratively update the weights.

[0065] The method for autonomous decision-making of defects for industrial inspection in the embodiment of the present application can collect the device operation state information of multiple modalities of industrial equipment through multiple spectral image signals by using a multi-spectral fusion terminal. Then, a cross-modal depth analysis model built in the cloud server is used to establish the features and channel fusion of the above-mentioned multiple modalities of device operation state information. Specifically, a two-stream heterogeneous deep network framework is used to perform cross-modal learning on the above-mentioned device operation state information, learn the joint distribution of the visual-text features of the industrial equipment, and then obtain the defect feature vector of the industrial equipment, so as to realize the utilization of cross-modal and cross-dimensional information; then, according to the hybrid dual-engine reasoning mode, multi-dimensional matching is performed between the defect feature vector and similar cases in the maintenance knowledge base, and the disposal plan is screened. Combining the matched cases and disposal plans can automatically obtain the maintenance strategy for the defect features of the industrial equipment, and the decision-making process does not need to rely on technicians. Finally, the defect information and maintenance strategy corresponding to the defect feature vector are sent to the communication terminal of the maintenance personnel, and the maintenance of industrial equipment defects can be realized. Through the above solution, the problem that multi-modal data is fragmented in the prior art and the feature-level fusion cannot be effectively established, resulting in low cross-modal information utilization rate, can be solved. Furthermore, the full process automation of device defect detection, autonomous decision-making of maintenance strategies, and device maintenance disposal can be realized.

[0066] In addition, it should be noted that in the existing maintenance knowledge base, the knowledge evolution is lagging, and the update of the maintenance case base depends on manual input, making it impossible to achieve a closed-loop iteration from on-site feedback to knowledge refinement. To solve the above problems, in the embodiments of the present application, when the improvement rate of industrial equipment is less than a certain level, necessary model updates are triggered. The daily maintenance score updates the weights of the graph nodes through knowledge distillation to form a closed-loop feedback, and the system iteratively updates the weights.

[0067] Specifically, as a preferred embodiment, as Figure 8 shown, after the step S140 of the above device defect autonomous decision-making method: sending the defect information and maintenance strategy corresponding to the defect feature vector to the communication terminal of the maintenance personnel, it further includes: S150: Obtain the maintenance score signal of the industrial equipment; S160: Adopt the knowledge distillation technology, and use the maintenance score signal to adjust the node weights of the corresponding dynamic knowledge graph of the maintenance knowledge base, and establish a mapping relationship between the score and the weight; S170: Optimize the maintenance knowledge base using the mapping relationship between the score and the weight.

[0068] Knowledge distillation is a technology that transfers the knowledge of a large model (teacher model) to a small model (student model). The goal is to train the student model to achieve or approach the performance of the teacher model at a lower computational cost. Specifically, in combination with Figure 14 shown, the present application first extracts features from the maintenance score signal; then constructs a teacher model, whose input is node embeddings and multi-dimensional score features, and the output is the probability distribution of node weights; and establishes a multi-source score fusion loss function, a score-weight consistency loss function, and a distillation optimization objective. Finally, based on the teacher model, a mapping relationship between the score and the weight is generated to update the maintenance knowledge base.

[0069] In summary, for the technical solution provided by the embodiments of the present application, after repairing the defects of industrial equipment, it is also necessary to feedback the results and optimize the model to achieve closed-loop feedback. For example, after the maintenance personnel receive the maintenance decision and perform the maintenance, they will continuously collect the equipment operation parameters (such as temperature and vibration values) to calculate the equipment health index (HI). When the HI improvement rate < 20%, an analysis of the maintenance cases in the maintenance knowledge base is triggered, the maintenance decision is manually corrected, and the model parameters of the cross-modal depth analysis model are periodically fine-tuned, such as the parameters related to the convolution kernel.

[0070] In addition, referring to Figure 9 , Figure 9 is a schematic flowchart of a method for defect autonomous decision-making for industrial inspection provided by the embodiments of the present application. As Figure 9 shown, the method for autonomous decision-making of the device defect includes: S201: Multimodal image acquisition.

[0071] S202: Cross-modal depth analysis model.

[0072] S203: Dynamic knowledge decision-making system.

[0073] S204: Determine whether there is a device defect; if so, execute step S205; if not, return to execute step S201.

[0074] S205: Match the maintenance knowledge base.

[0075] S206: Issue a maintenance instruction.

[0076] In summary, the defects of the existing technology stem from three major architectural contradictions: First, multi-modal perception is fragmented, and visible light, infrared, and ultraviolet imaging data and sensor parameters are analyzed independently, lacking a cross-modal feature correlation model; second, knowledge evolution is lagging, and the update of the maintenance case base depends on manual input, and it is impossible to achieve a closed-loop iteration from on-site feedback to knowledge refinement; third, the decision-making process is black-box, although the deep learning model can output the defect classification result, but lacks an interpretable maintenance strategy derivation link and lacks a knowledge matching process. The defect autonomous decision-making solution for industrial inspection provided in the above embodiments of the present application, the proposed "terminal-cloud-library" method, can integrate terminal collection, cloud retrieval, knowledge base matching and knowledge recommendation, and effectively solve the above three major architectural contradictions existing in the existing methods.

[0077] In addition, refer to Figure 10 , Figure 10 A defect autonomous decision-making device provided in an embodiment of the present application includes: An information collection module 110, configured to use a multi-spectral fusion terminal to collect device operation status information of an industrial device and upload it to a cloud server; A cross-modal learning module 120, configured to use a cross-modal depth analysis model built in the cloud server to perform cross-modal learning on the device operation status information according to a dual-stream heterogeneous deep network architecture, learn the joint distribution of visual-text features of the industrial device, and obtain a defect feature vector of the industrial device; A strategy learning module 130, configured to perform multi-dimensional matching on the defect feature vector and similar cases in the maintenance knowledge base according to a hybrid dual-engine reasoning mode and screen a disposal solution corresponding to the defect feature vector to obtain a maintenance strategy; An information distribution module 140, configured to send the defect information corresponding to the defect feature vector and the maintenance strategy to a communication terminal of a maintenance personnel.

[0078] The embodiment of the present application is implemented through the following technical solutions: inspection personnel collect equipment image data through handheld terminals; equipment image data is uploaded to a large cloud model, and defect information is detected in the cloud; the maintenance knowledge base retrieves similar cases, matches maintenance strategies, and generates disposal plans; defect information and maintenance strategies are sent to maintenance personnel for maintenance.

[0079] In summary, the technical solutions provided by the above embodiments of the present application address the three major technical problems that have long existed in the field of industrial inspections, namely, multimodal data fragmentation analysis, static solidification of maintenance knowledge, and excessive reliance on manual labor in the decision-making process. A method for autonomous decision-making on equipment defects based on the "end-cloud-library" collaborative architecture is proposed. By constructing a multimodal feature fusion engine and a dynamic knowledge evolution system, the performance bottleneck of traditional technologies is broken through, and the full process from defect perception to maintenance decision-making is realized to be autonomous, solving the problem of excessive reliance on manual experience by traditional methods. After actual verification, the time required to generate maintenance plans is compressed from hours to minutes.

[0080] In addition, it should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0081] In addition, see Figure 11 The electronic device provided by the present application includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the defect autonomous decision-making method for industrial inspection of any of the above embodiments.

[0082] Reference below Figure 11 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic devices in the embodiments of the present application can include but are not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions: tablet computers), PMPs (Portable Media Players: portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 11 The device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0083] like Figure 11As shown, the electronic device can include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM: Read Only Memory) 1002 or the program loaded from the storage device 1003 into the random access memory (RAM: Random Access Memory) 1004. In the RAM 1004, various programs and data required for the operation of the above-mentioned electronic device are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. The input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can operate to wirelessly or wirelesly communicate with other devices for data exchange with the defect autonomous decision-making device for industrial inspection. Although the figure shows a model construction device with various systems, it should be understood that it is not required to implement or have all the shown systems. More or fewer systems can be alternatively implemented or had.

[0084] Specifically, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program codes for executing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of the embodiments disclosed in the present application are executed.

[0085] It should be understood that the various parts disclosed in the present application can be implemented by hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in a suitable manner in any one or more embodiments or examples.

[0086] As described above, it is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims described above.

[0087] The present application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the defect autonomous decision-making method for industrial inspection in the above-mentioned embodiments.

[0088] The computer-readable storage medium provided by the present application can be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory, optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this embodiment, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, device, or component. The program code contained on the computer-readable storage medium can be transmitted by any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0089] The above computer-readable storage medium carries one or more programs which, when executed by a model construction device, can write computer program code for performing the operations of the present application in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, by connecting through the Internet using an Internet service provider).

[0090] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagram can represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks can occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0091] The modules involved in the embodiments described in the present application can be implemented in software or in hardware. Among them, the name of the module does not constitute a limitation to the unit itself in some cases.

[0092] The present application also provides a computer program product, including a computer program which, when executed by a processor, implements the steps of the above-mentioned defect autonomous decision-making method for industrial inspection.

[0093] Each embodiment in this specification is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments.

[0094] The above description is only for the embodiments of the present application and is not intended to limit the present application. For those skilled in the art, various changes and modifications can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the protection scope of the present application.

Claims

1. A defect autonomous decision-making method for industrial inspection tours, characterized in that, Including: Using a multispectral fusion terminal to collect the device operation status information of industrial equipment and upload it to a cloud server; Using a cross-modal depth analysis model built in the cloud server, performing cross-modal learning on the device operation status information according to a dual-stream heterogeneous depth network architecture, learning the joint distribution of visual-text features of the industrial equipment, and obtaining a defect feature vector of the industrial equipment; According to a hybrid dual-engine inference mode, performing multi-dimensional matching on the defect feature vector and similar cases in a maintenance knowledge base and screening a disposal plan corresponding to the defect feature vector to obtain a maintenance strategy; Sending the defect information corresponding to the defect feature vector and the maintenance strategy to the communication terminal of maintenance personnel.

2. The method according to claim 1, wherein After the step of sending the defect information corresponding to the defect feature vector and the maintenance strategy to the communication terminal of maintenance personnel, the method further includes: Obtaining a maintenance scoring signal of the industrial equipment; Adopting a knowledge distillation technique, using the maintenance scoring signal to adjust the node weights of a dynamic knowledge graph corresponding to the maintenance knowledge base, and establishing a mapping relationship between the score and the weight; Optimizing the maintenance knowledge base using the mapping relationship between the score and the weight.

3. The method according to claim 1, wherein The step of using a multispectral fusion terminal to collect the device operation status information of industrial equipment and upload it to a cloud server includes: Using the multispectral fusion terminal to collect a multispectral status image of the industrial equipment; Using a 3D contour calibration algorithm to compare in real time the spatial coordinates of the industrial equipment in the multispectral status image with the corresponding spatial features in a preset CAD model; Combining a gyroscope attitude compensation mechanism to perform spatial coordinate calibration on the spatial features of the industrial equipment; Performing filtering and denoising processing on the multispectral status image after spatial coordinate calibration to obtain the device operation status information of the industrial equipment.

4. The method according to claim 1, wherein The step of using a cross-modal depth analysis model built in the cloud server to perform cross-modal learning on the device operation status information according to a dual-stream heterogeneous depth network architecture, learning the joint distribution of visual-text features of the industrial equipment, and obtaining a defect feature vector of the industrial equipment includes: Uploading the device operation status information of the industrial equipment to the cross-modal depth analysis model of the cloud server, where the cross-modal depth analysis model has the dual-stream heterogeneous depth network architecture, and the device operation status information includes a multispectral status image and device status data; Controlling the visual processing stream of the dual-stream heterogeneous depth network architecture, using an object detection algorithm combined with a multi-scale attention mechanism to establish feature associations of the multispectral status image in dimensions of multiple scales, and detecting visual features of the multispectral status image; Controlling the text processing stream of the dual-stream heterogeneous depth network architecture, using a large text processing model to construct a domain semantic parsing dictionary, and using the domain semantic parsing dictionary to parse text features corresponding to the device status data; Using a cross-modal adversarial distillation pipeline to adversarially collect the visual features and text features; The generator network constructed using the cross-modal adversarial distillation pipeline learns the joint distribution of the visual features and text features according to the alignment loss function to obtain the defect feature vector.

5. The method according to claim 4, wherein The step of controlling the visual processing stream of the dual-stream heterogeneous deep network architecture, using an object detection algorithm combined with a multi-scale attention mechanism to establish feature associations of the multi-spectral status image in dimensions of multiple scales and detecting the visual features of the multi-spectral status image includes: Controlling the visual processing stream to adopt an improved ViT-Transformer algorithm, introducing a deformable convolutional layer to dynamically perceive the convolutional kernel offset of the multi-spectral status image, and capturing device defects in the multi-spectral status image; Introducing a multi-scale attention mechanism in the visual processing stream to establish feature associations of the multi-spectral status image in dimensions of multiple scales and identifying fuzzy defects of the industrial device; Obtaining the visual features corresponding to the image defects and the fuzzy defects.

6. The method according to claim 1, wherein The step of, according to the hybrid dual-engine inference mode, performing multi-dimensional matching on the defect feature vector and similar cases in the maintenance knowledge base and screening the disposal solutions corresponding to the defect feature vector to obtain a maintenance strategy includes: Invoking the dynamic knowledge graph corresponding to the maintenance knowledge base to perform multi-dimensional matching on the defect feature vector and similar cases, and calculating the cosine similarity between the defect feature and the similar cases according to the degree of multi-dimensional matching; Extracting maintenance cases with the cosine similarity greater than the similarity threshold; And Constructing an expert experience decision tree based on fuzzy logic, and using the expert experience decision tree to process the uncertain conditions corresponding to the maintenance cases through a fuzzy membership function to obtain disposal solutions; Integrating the maintenance cases and disposal solutions to obtain a maintenance strategy.

7. The method according to claim 1, characterized in that After the step of sending the defect information corresponding to the defect feature vector and the maintenance strategy to the communication terminal of the maintenance personnel, the method further includes: Continuously collecting the operating parameters of the industrial device after the industrial device is repaired; Inputting the operating parameters into a health index model to obtain the improvement rate of the industrial device after repair; Using the improvement rate to evaluate the maintenance disposal effect of the maintenance personnel; When the improvement rate is less than or equal to a predetermined improvement threshold, triggering a parameter update mechanism of the cross-modal depth analysis model; Obtaining the maintenance scoring signal uploaded by the communication terminal; Using the maintenance scoring signal to iteratively update the node weights of the dynamic knowledge graph corresponding to the maintenance knowledge base through knowledge distillation technology.

8. A defect autonomous decision-making device for industrial inspection tours, characterized in that, Including: An information collection module for using a multi-spectral fusion terminal to collect the device operating status information of an industrial device and uploading it to a cloud server; A cross-modal learning module for using the cross-modal depth analysis model built in the cloud server to perform cross-modal learning on the device operating status information according to the dual-stream heterogeneous deep network architecture, learning the joint distribution of the visual-text features of the industrial device, and obtaining the defect feature vector of the industrial device; A strategy learning module, configured to perform multi-dimensional matching on the defect feature vector and similar cases in the maintenance knowledge base according to the hybrid dual-engine inference mode, screen a disposal solution corresponding to the defect feature vector, and obtain a maintenance strategy; An information distribution module, configured to distribute the defect information corresponding to the defect feature vector and the maintenance strategy to the communication terminals of maintenance personnel.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the defect autonomous decision-making method for industrial inspection as described in any one of claims 1 to 7.

10. A computer storage medium having computer-executable instructions stored thereon, characterized in that, When the computer-executable instructions are executed by the processor, they implement the defect autonomous decision-making method for industrial inspection as described in any one of claims 1 to 7.

Images