Intelligent device inspection method and device fusing robot vision, and robot

Abstract

The application relates to an intelligent device inspection method and device fusing robot vision and a robot. The method comprises the following steps: acquiring visible image data, infrared image data and multi-source sensing data of an intelligent device; performing time synchronization and feature fusion on the visible image data, the infrared image data and the multi-source sensing data to obtain a fusion feature vector; extracting hidden defects of the fusion feature vector based on a convolutional neural network to obtain a hidden defect category; and obtaining an inspection report of the hidden defects according to the hidden defect category based on a Bayesian network algorithm; the inspection report comprises the hidden defect category and a defect reason. The method can reversely deduce a defect forming reason through visual data, and provide a time-sensitive suggestion for product maintenance management decision-making.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of product maintenance management, and particularly relates to a method and device for intelligent equipment inspection integrated with robot vision and a robot. BACKGROUND

[0002] With the development of product maintenance management technology and industrial automation, intelligent equipment inspection technology based on multi-modal robot vision has the characteristics of high-precision defect detection, real-time data acquisition and cloud collaborative management, thereby promoting the current vision-driven intelligent equipment health monitoring method.

[0003] In the current vision-driven intelligent equipment health monitoring method, the surface image of the equipment is collected by the vision sensor (such as a high-resolution camera, an infrared thermal imager) carried by the inspection robot, and a deep learning model (such as YOLO, U-Net) is used to realize the positioning and classification of cracks, corrosion and other obvious defects. Some methods further introduce vibration or temperature sensor data to improve detection robustness.

[0004] However, the current vision-driven intelligent equipment health monitoring method can only identify surface defects (such as crack length and rust area) of the equipment, but cannot infer the root cause of the defects from the visual data, resulting in a lag in product maintenance management decision-making. SUMMARY

[0005] Therefore, it is necessary to provide a method and device for intelligent equipment inspection integrated with robot vision, which can identify hidden defects and their causes.

[0006] In a first aspect, the application provides a method for intelligent equipment inspection integrated with robot vision, comprising:

[0007] obtaining visible image data, infrared image data and multi-source sensor data of the intelligent equipment;

[0008] time-synchronizing and feature-fusing the visible image data, the infrared image data and the multi-source sensor data to obtain a fusion feature vector;

[0009] extracting hidden defects of the fusion feature vector based on a convolutional neural network to obtain a hidden defect category;

[0010] obtaining a hidden defect inspection report according to the hidden defect category based on a Bayesian network algorithm; the inspection report includes the hidden defect category and a defect cause.

[0011] Further, obtaining a hidden defect inspection report according to the hidden defect category based on a Bayesian network algorithm; the inspection report includes the hidden defect category and a defect cause, including:

[0012] According to the hidden defect category, match the preset defect element knowledge base, and obtain a dynamic defect causal graph;

[0013] Based on the Bayesian network algorithm, the reason probability of causing the defect in the dynamic defect causal graph is calculated, and a defect root cause probability distribution is obtained;

[0014] Based on the natural language model, the defect root cause probability distribution is converted into a text report, and an inspection report of the hidden defect is obtained.

[0015] Further, according to the hidden defect category, match the preset defect element knowledge base, and obtain a dynamic defect causal graph, including:

[0016] Using the following formula, according to the hidden defect category, match the defect element knowledge base, and obtain the corresponding defect reason data:

[0017]

[0018] Wherein, Rule(C d ) is the defect reason data, X i is the i-th defect reason in the defect element knowledge base, w i is the causal weight of the i-th defect reason, C d is the hidden defect category, is the defect reason set in the defect element knowledge base;

[0019] Using the following formula, combined with the running parameters of the intelligent device, the defect reason data is expanded, and dynamic defect causal data is obtained:

[0020] G(V,E)=Rule(C d )∪(p j →X i )

[0021] Wherein, G(V,E) is the dynamic defect causal data, p j is the j-th running parameter of the intelligent device, and X i is the i-th defect reason in the defect element knowledge base;

[0022] The dynamic defect causal data is combined to obtain a dynamic defect causal graph.

[0023] Further, based on the Bayesian network algorithm, the reason probability of causing the defect in the dynamic defect causal graph is calculated, and a defect root cause probability distribution is obtained, including:

[0024] Using the following formula, the reason probability of causing the defect in the dynamic defect causal graph is calculated, and the defect root cause probability is obtained:

[0025]

[0026] Wherein, P(Xi = 1 | Pa(X i )) is a defect root cause probability, Pa(X i ) is a set of intelligent device operating parameters in dynamic defect causal data, σ is an activation function, β j is a regression coefficient, p j is the influence coefficient of the jth operating parameter of the intelligent device, and γ is a bias term.

[0027] The defect root cause probability is iteratively sampled for a preset number of times to obtain a defect root cause probability distribution.

[0028] Further, based on the natural language model, the defect root cause probability distribution is converted into a text report to obtain an inspection report of the hidden defect, including:

[0029] Using the following formula, based on the extension of the Shapley value in game theory, the probability distribution contribution in the defect root cause probability distribution is calculated:

[0030]

[0031] Wherein, φ j is the probability distribution contribution of the jth intelligent device operating parameter, S is a subset of intelligent device operating parameters contained in the defect root cause probability distribution, N is a set of intelligent device operating parameters, and f(S) is the defect root cause probability distribution of the subset of intelligent device operating parameters.

[0032] The probability distribution contribution and the intelligent device operating parameter corresponding to the probability distribution contribution are combined to obtain a key factor list.

[0033] The key factor list and the hidden defect category are traversed, and a preliminary text sequence is generated according to a preset sorting rule.

[0034] Using the following formula, the cross-entropy loss is calculated according to the key factor list:

[0035]

[0036] Wherein, is the cross-entropy loss, T is the total number of words in the preliminary text sequence, w t is the tth word of the preliminary text sequence, is the key factor list.

[0037] Based on the natural language model, the cross-entropy loss is adjusted according to the cross-entropy loss to obtain an inspection report of the hidden defect.

[0038] Further, the visible image data, the infrared image data and the multi-source sensing data are time-synchronized and feature-fused to obtain a fusion feature vector, including:

[0039] The time difference values of the visible image data and the infrared image data, the multi-source sensing data, and the time difference values of the infrared image data and the multi-source sensing data are calculated, the time difference values are compared, and a maximum time deviation is obtained;

[0040] According to the maximum time deviation, a maximum offset is calculated using the following formula:

[0041]

[0042] wherein τ max is the maximum offset, is a function of selecting the maximum value of τ, N is the window length, s recv (n) is a received value of a received signal, s ref (n-τ) is a reference value of a reference signal;

[0043] According to the maximum offset, a corrected timestamp of the visible image data, the infrared image data and the multi-source sensing data is calculated using the following formula:

[0044] δt=τ max ·T s

[0045] wherein δt is the corrected timestamp of the data, τ max is the maximum offset, T s is a sampling period of the data;

[0046] The corrected timestamp is combined with the visible image data, the infrared image data and the multi-source sensing data to obtain corresponding synchronous visible image data, synchronous infrared image data and synchronous multi-source sensing data;

[0047] If the time interpolation of the synchronous multi-source sensing data is greater than a preset threshold, the multi-source sensing data is completed by using an interpolation algorithm through the following formula to obtain updated synchronous multi-source sensing data:

[0048]

[0049] wherein t is a timestamp of data loss, S(t) is the completed data at the timestamp t, α is an adjustment parameter, S(t2) is the data at the next sampling timestamp, S(t1) is the data at the previous sampling timestamp, t2 is the next sampling timestamp, t1 is the previous sampling timestamp, S l (t) is a linear completion term, S m (t) is a model compensation term, K is the number of physical model terms, A k is a data transmission amplitude, β is an attenuation coefficient, f k is a dominant frequency of the data signal, φ k is a data transmission phase;

[0050] Extract the morphological distribution characteristics of the synchronous infrared image data to obtain synchronous infrared feature data;

[0051] Fuse the synchronous visible image data, the synchronous infrared feature data and the synchronous multi-source sensing data to obtain a fusion feature vector.

[0052] Further, the synchronous visible image data, the synchronous infrared feature data and the synchronous multi-source sensing data are fused to obtain a fusion feature vector, including:

[0053] According to a preset segmentation region size, the synchronous visible image data is segmented to obtain a visible image block set;

[0054] The following formula is used to calculate the gray value pixel data of the visible image block set:

[0055]

[0056] wherein h(k) is the number of pixels with a gray value of k, Tile(m, n) is a preset segmentation region, δ is a statistical function, and I(x, y) is a gray value of a coordinate (x, y) in the synchronous visible image data;

[0057] The gray value pixel data is combined to obtain an initial gray histogram set;

[0058] According to a preset gray value quantity threshold, the initial gray histogram set is limited to obtain a cropped gray histogram set. The following formula is used to map the cropped gray histogram set to obtain a new gray value:

[0059]

[0060] wherein k ′ is a new gray value obtained by mapping a gray value k in the initial gray histogram set, f is an integer function, K is a number of gray levels in the initial gray histogram set, h ′ (k) is the number of pixels with a gray value of k in the cropped gray histogram set;

[0061] According to the new gray value, the initial gray histogram set is mapped to obtain a new gray histogram set;

[0062] The following formula is used to perform bilinear interpolation on the new gray histogram set to obtain enhanced synchronous image data:

[0063]

[0064] wherein I c is the enhanced synchronous image data, w ij is a neighboring weight of a coordinate (i, j), is a new gray histogram set of the segmented region with coordinates (i, j), and I(i, j) is a gray value with coordinates (i, j) in the synchronous visible image data;

[0065] The principal component analysis method is adopted to perform principal component feature extraction and feature dimension reduction processing on the enhanced synchronous image data, to obtain reserved feature image data.

[0066] The reserved feature image data, the synchronous infrared feature data and the synchronous multi-source sensing data are weighted and fused through the attention mechanism network, to obtain a fusion feature vector.

[0067] Further, based on the convolutional neural network, hidden defects of the fusion feature vector are extracted, to obtain a hidden defect category, including:

[0068] The fusion feature vector is converted into a two-dimensional structure, to obtain a two-dimensional fusion feature map.

[0069] Based on the convolutional neural network, the two-dimensional fusion feature map is input into a defect classification model, to obtain a hidden classification category.

[0070] The defect classification model includes:

[0071] An input layer is configured to input the two-dimensional fusion feature map.

[0072] A multi-scale convolution layer is configured to extract local features and intermediate features of the two-dimensional fusion feature map, to obtain a defect feature map, and an output of each layer is input into a next layer; a convolution output formula in the multi-scale convolution layer is: wherein is an output of an lth multi-scale convolution layer, ReLU is an activation function, is a convolution kernel of the lth multi-scale convolution layer, is an output of an (l-1)th multi-scale convolution layer, b (l) is a bias term of the lth multi-scale convolution layer;

[0073] A pooling layer is configured to compress the defect feature map, to obtain a compressed defect feature map; an output formula of the pooling layer is: wherein, P i,j is the compressed defect feature map, is a maximum value operation, is an output of the multi-scale convolution layer;

[0074] A full connection layer is configured to determine a defect category of the compressed defect feature map, to obtain a hidden defect category.

[0075] An output layer is configured to output the hidden defect category.

[0076] In a second aspect, the application further provides an intelligent device inspection device fused with robot vision, including:

[0077] an information acquisition module, configured to acquire visible image data, infrared image data and multi-source sensing data of the intelligent device;

[0078] an information fusion module, configured to perform time synchronization and feature fusion on the visible image data, the infrared image data and the multi-source sensing data to obtain a fusion feature vector;

[0079] a defect extraction module, configured to extract hidden defects of the fusion feature vector based on a convolutional neural network to obtain a hidden defect category;

[0080] a report generation module, configured to obtain an inspection report of the hidden defects according to the hidden defect category based on a Bayesian network algorithm; the inspection report includes the hidden defect category and a defect reason.

[0081] In a third aspect, the present application further provides a robot, comprising a memory, a processor and a sensor, wherein the memory stores a computer program, and the processor implements any one of the intelligent device inspection methods of fusing robot vision according to the computer program.

[0082] The intelligent device inspection method of fusing robot vision provided by the embodiments of the present application, the device inspection device of fusing robot vision and the robot, by acquiring visible image data, infrared image data and multi-source sensing data of the intelligent device; the visible image data, the infrared image data and the multi-source sensing data are time synchronized and feature fused to obtain a fusion feature vector; based on a convolutional neural network, hidden defects of the fusion feature vector are extracted to obtain a hidden defect category; based on a Bayesian network algorithm, an inspection report of the hidden defects is obtained according to the hidden defect category; the inspection report includes the hidden defect category and a defect reason. The combination of visible visual data, infrared visual data and device detection sensing data is realized, defect detail features are extracted, the defect type is obtained through the defect detail features, and the reason for forming the defect is obtained by reverse deduction, which provides time-effective suggestions for product maintenance management decision-making. BRIEF DESCRIPTION OF DRAWINGS

[0083] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technical solutions, the following will briefly introduce the drawings needed to be used in the embodiment or related technical description. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creating any creative labor.

[0084] Figure 1 A flowchart of an intelligent device inspection method of fusing robot vision is provided for an exemplary embodiment of the present application;

[0085] Figure 2A structural block diagram of an intelligent device inspection device fusing robot vision is provided for an exemplary embodiment of the present application; DETAILED DESCRIPTION

[0086] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0087] In one embodiment, as shown in Figure 1 A robot vision-fused intelligent device inspection method is provided, and the present embodiment is exemplified by the method applied to a robot. It should be understood that the method can also be realized by interaction between the robot and the server or interaction between the robot and the terminal. In the present embodiment, the method comprises S101-S104, wherein:

[0088] S101, acquiring visible image data, infrared image data and multi-source sensing data of the intelligent device.

[0089] Specifically, the robot acquires visible image data, infrared image data and multi-source sensing data of the intelligent device. Illustratively, the visible image data and the infrared image data are obtained by a high-resolution visible light camera and an infrared thermal imager carried by the robot. Alternatively, the multi-source sensing data can be detected by a data sensor carried by the robot, or can be input by the user, and the multi-source sensing data includes but is not limited to vibration data, temperature data, humidity data and current sensing data.

[0090] S102, time synchronization and feature fusion of the visible image data, the infrared image data and the multi-source sensing data are performed to obtain a fusion feature vector.

[0091] Specifically, the robot adopts PTP protocol to perform time synchronization processing on the visible image data, the infrared image data and the multi-source sensing data, and performs weighted fusion on the visible image data, the infrared image data and the multi-source sensing data that have undergone time synchronization processing through an attention mechanism network to obtain a fusion feature vector. Illustratively, the robot calculates the weighted weight of the visible image data, the infrared image data and the multi-source sensing data through the attention mechanism network, and fuses the visible image data, the infrared image data and the multi-source sensing data according to the weighted weight to obtain the fusion feature vector.

[0092] S103, based on a convolutional neural network, hidden defects of the fusion feature vector are extracted to obtain a hidden defect category.

[0093] Specifically, based on the convolutional neural network, the robot obtains the hidden defect category corresponding to the fusion feature vector according to the fusion feature vector. Illustratively, the number of hidden defect categories can be set according to the user's demand for intelligent device maintenance, or can be obtained by training the generative adversarial network.

[0094] In S104, based on the Bayesian network algorithm, a hidden defect inspection report is obtained according to the hidden defect category. The inspection report includes the hidden defect category and the defect reason.

[0095] Specifically, based on the Bayesian network algorithm, the robot obtains the reasons that may cause the defect category, constructs the causal relationship, calculates the probability corresponding to each causal relationship, combines the natural language model, converts the reasoning result into a human-readable inspection report, and obtains the hidden defect inspection report. Illustratively, the natural language model can be GPT. Alternatively, the hidden defect inspection report includes but is not limited to the hidden defect category, the identified probability, the reason for causing the defect, and the probability of the reason for causing the defect.

[0096] In this embodiment, by obtaining the visible light image, the infrared image and the multi-source sensing data of the device, the fusion feature vector is obtained through time synchronization and cross-modal feature fusion. The convolutional neural network is used to extract the hidden defect category and its probability from the fusion feature. Through the Bayesian network and the natural language model, the causal relationship between the defect and the operating parameter is constructed, and the inspection report containing the root cause analysis of the hidden defect is generated. The intelligent device multi-modal feature extraction and analysis of the fusion robot vision data are realized, and the reason for the defect formation is deduced through the feature data, which provides time-effective suggestions for product maintenance management decision.

[0097] In one embodiment, based on the Bayesian network algorithm, a hidden defect inspection report is obtained according to the hidden defect category. The inspection report includes the hidden defect category and the defect reason, including:

[0098] In S201, according to the hidden defect category, a preset defect meta-knowledge base is matched to obtain a dynamic defect causal graph.

[0099] Specifically, the robot retrieves the associated causal rules and possible reasons from the preset defect knowledge base based on the identified hidden defect category, obtains all possible reasons that may cause the hidden defect category, and the reason and the hidden defect category form a dynamic defect causal graph. Illustratively, the preset defect meta-knowledge base can be collected by the user according to the type or characteristics of the intelligent device, including but not limited to the type of device defect, the reason for causing the defect, and the operating parameter of the intelligent device affecting the defect type.

[0100] In S202, based on the Bayesian network algorithm, the probability of the reason for causing the defect in the dynamic defect causal graph is calculated to obtain a defect root cause probability distribution.

[0101] Specifically, the robot calculates the posterior probability of each defect cause according to the aforementioned dynamic defect causal graph, utilizes the conditional probability propagation mechanism of the Bayesian network, and infers the probability contribution of each defect cause to obtain a defect root cause probability distribution.

[0102] S203, based on the natural language model, the defect root cause probability distribution is converted into a text report to obtain an inspection report of the hidden defect.

[0103] Specifically, the robot adopts a pre-trained natural language model to convert the defect root cause probability distribution obtained in S202 into a structured text report to obtain an inspection report of the hidden defect. Illustratively, the natural language model can generate a natural language description conforming to engineering semantics according to the input probability distribution and the operating parameters of the intelligent device through a decoder.

[0104] The embodiment matches a preset defect element knowledge base, combines the hidden defect causes and the operating parameters of the intelligent device to construct a dynamic defect causal graph, forms a network structure containing causal relationships and parameter nodes, calculates the posterior probability distribution of each cause based on the Bayesian network, and obtains the defect root cause probability distribution through sampling iteration optimization of probability stability. The natural language model is driven to combine the contribution degree of the calculated operating parameters of the intelligent device to generate an inspection report of the hidden defect, obtain the hidden defect category probability, the causes and the probability of causing the hidden defect category through reasoning calculation, and visually display the same, thereby providing a reference with timeliness and accuracy for the maintenance and daily use of the intelligent device.

[0105] In one of the embodiments, according to the hidden defect category, the preset defect element knowledge base is matched to obtain a dynamic defect causal graph, which includes:

[0106] S301, according to the hidden defect category, the defect element knowledge base is matched to obtain corresponding defect cause data using the following formula:

[0107]

[0108] wherein, Rule(C d ) is the defect cause data, X i is the i-th defect cause in the defect element knowledge base, w i is the causal weight of the i-th defect cause, C d is the hidden defect category, is the set of defect causes in the defect element knowledge base.

[0109] Specifically, the robot matches the information of the hidden defect category in the defect element knowledge base according to the hidden defect category to obtain corresponding defect cause data. Illustratively, the defect cause data Rule(C d() is a set of causes for this hidden defect type recorded in the defect meta-knowledge base. Optionally, the i-th defect cause X in the defect meta-knowledge base i These are the causes recorded in the defect meta-knowledge base that may lead to the hidden defect category obtained in S103. Schematic, the causal weight w of the i-th defect cause... i Used to indicate cause X i For hidden defect category C d The intensity of the impact can be set based on the maintenance history of the smart device. Optionally, the hidden defect category C d This refers to the hidden defect category obtained from S103. Indicatively, it represents the set of defect causes in the defect meta-knowledge base. It is the hidden defect category C recorded in the defect meta-knowledge base. d Reason X i A set of.

[0110] S302, using the following formula, combined with the operating parameters of the intelligent device, expand the defect cause data to obtain dynamic defect cause-effect data:

[0111] G(V,E)=Rule(C d )∪(p j →X i )

[0112] Where G(V,E) is dynamic defect causal data, p j It is the j-th operating parameter of the smart device, x i It is the i-th defect cause in the defect meta-knowledge base.

[0113] Specifically, the robot incorporates the influence of the operating parameters of intelligent devices on the categories of hidden defects into the defect cause data, obtaining dynamic defect causal data. Illustratively, the dynamic defect causal data G(V,E) includes... Optionally, Rule(C... d ) is the defect cause data obtained from S301. Indicatively, the j-th operating parameter p of the intelligent device... j It is recorded in the defect meta-knowledge base and may cause the i-th defect cause X in the defect meta-knowledge base. i The operating parameters. Optionally, the i-th defect cause X in the defect meta-knowledge base. i It is recorded in the defect meta-knowledge base and may be the reason for the hidden defect category obtained in S103.

[0114] S303, dynamic defect causal data combination to obtain dynamic defect causal graph.

[0115] Specifically, the robot combines the expanded dynamic defect causal data according to a graph structure to obtain a complete dynamic defect causal graph. Schematic, the nodes of this causal graph include, but are not limited to, hidden defect types, causes of defects, and intelligent device operating parameters; the edges of the causal graph include, but are not limited to, influence weights and conditional probabilities.

[0116] This embodiment extracts causal rules and weights associated with the current defect category by matching the defect knowledge base, and generates corresponding defect cause data. It also dynamically expands the causal data by combining the real-time operating parameters of the equipment, adds the parameters as observation nodes to the causal graph, and establishes the association edges between them and the defect causes. It integrates all nodes and edges and combines them according to the graph structure to obtain a complete dynamic defect causal graph, thereby improving the accuracy of root cause analysis.

[0117] In one embodiment, based on a Bayesian network algorithm, the probability of causes of defects in a dynamic defect cause-effect graph is calculated to obtain a defect root cause probability distribution, including:

[0118] S401, use the following formula to calculate the probability of a cause of a defect in the dynamic defect cause-effect graph, and obtain the root cause probability of the defect:

[0119]

[0120] Wherein, P(X) i =1|Pa(X) i Pa(X) is the probability of the root cause of the defect. i ) is the set of intelligent device operating parameters in dynamic defect causal data, σ is the activation function, and β is the activation parameter. j It is the regression coefficient, p j γ is the influence coefficient of the j-th operating parameter of the intelligent device, and γ is the bias term.

[0121] Specifically, based on the dynamic defect cause-effect graph, the root cause probability formula is used to calculate the defect root cause probability. Illustratively, the defect root cause probability P(X) is... i =1|Pa(X) i The probability of the root cause of the hidden defect category after activation is represented by )). Optionally, the set of intelligent device operating parameters Pa(X) in the dynamic defect causal data... i () is the dynamic defect cause X obtained from S303, which is the possible cause of the defect. i The set of operating parameters of the affected smart devices. Schematic, the activation function σ can be the sigmoid function. Optionally, the regression coefficient β... j The influence coefficient p used to reflect the j-th operating parameter of the intelligent device j For the i-th defect cause X i The strength of the influence. Schematally, the influence coefficient p of the j-th operating parameter of the intelligent device.j is the jth operating parameter in the operating parameter set Pa(X i ) of the intelligent device. Optionally, the bias term γ is used to adjust the baseline probability, such as the default defect occurrence probability.

[0122] S402, iteratively sample the defect root cause probability for a preset number of times to obtain a defect root cause probability distribution.

[0123] Specifically, the Bayesian network is iteratively sampled to estimate the posterior probability distribution, eliminate random noise and converge to a steady-state distribution. When the statistics (such as mean, variance) of the sampling sequence is stable, the sampling is stopped, and the defect root cause probability distribution is obtained. Illustratively, the Markov Chain Monte Carlo method can be used for multiple sampling. Optionally, the number of samplings can be set according to the user's requirements for defect categories and causing probabilities.

[0124] The embodiment uses the Bayesian network algorithm to weight and sum the operating parameters in the dynamic causal diagram through the logistic regression model and map them to the defect cause probability, iteratively samples and updates the defect root cause probability, and obtains the defect root cause probability distribution by iteratively updating the node state and statistically converging the probability distribution. The reason contribution degree is intuitively quantified, and the single probability calculation bias is reduced.

[0125] In one embodiment, based on the natural language model, the defect root cause probability distribution is converted into a text report to obtain an inspection report of the hidden defect, including:

[0126] S501, based on the extension of the Shapley value in game theory, the probability distribution contribution in the defect root cause probability distribution is calculated using the following formula:

[0127]

[0128] where φ j is the probability distribution contribution of the jth intelligent device operating parameter, S is the subset of intelligent device operating parameters contained in the defect root cause probability distribution, N is the set of intelligent device operating parameters, and f(S) is the defect root cause probability distribution of the subset of intelligent device operating parameters.

[0129] Specifically, based on the extension of the Shapley value in game theory, the intelligent device operating parameters contained in the defect root cause probability distribution are calculated to contribute to the probability distribution of causing the hidden defect category. Illustratively, the jth key factor φ jis the Shapley value of the jth, which is used to quantify its contribution to the defect probability. Optionally, the set of intelligent device operating parameters N is a set of all intelligent device operating parameters, which can be recorded in the defect element knowledge base. Illustratively, the subset of intelligent device operating parameters S contained in the defect root cause probability distribution is any subset of the set of intelligent device operating parameters N, which is used to traverse all possible subset combinations. Optionally, the defect root cause probability distribution f(S) of the subset of intelligent device operating parameters is the defect root cause probability distribution obtained by S402.

[0130] S502, combining the probability distribution contribution and the intelligent device operating parameter combination corresponding to the probability distribution contribution to obtain a key factor list.

[0131] Specifically, according to the probability distribution contribution obtained by S501, the probability distribution contribution and the intelligent device operating parameter combination corresponding to the probability distribution contribution are combined to obtain a key factor list. Illustratively, it can be combined in the order of the size of the probability distribution, which provides a more logical data source for combining text based on a natural language model.

[0132] S503, traversing the key factor list and the hidden defect category, and generating a preliminary text series according to a preset sorting rule.

[0133] Specifically, according to the key factor list and the hidden defect category, the preliminary text series is generated by filling according to a preset sorting rule. Illustratively, the preset rule can be: such as "the defect may be caused by {key factor 1} (corresponding contribution {value 1}), {key factor 2} (corresponding contribution {value 2}), …, {key factor n} (corresponding contribution {value n})", wherein the setting of n can be set according to the user's demand for the number of defect reason reasoning.

[0134] S504, using the following formula to calculate the cross-entropy loss according to the key factor list:

[0135]

[0136] wherein, is the cross-entropy loss, T is the total number of words in the preliminary text sequence, w t the tthword of the preliminary text sequence, is the key factor list.

[0137] Specifically, the cross-entropy loss is calculated according to the cross-entropy loss calculation formula, and the fluency and logicality of the generated text are optimized. Optionally, the cross-entropy loss is used to measure the difference between the generated text and the true report. Optionally, the total number of words T of the preliminary text sequence is the total number of words of the preliminary text sequence generated according to the key factor list and the hidden defect category according to a preset sorting rule. Illustratively, the t-th word w t is any one word in the generated preliminary text sequence, used to traverse the preliminary text sequence. Optionally, the key factor list is the key factor list obtained in S501.

[0138] S505, based on the natural language model, adjusts the preliminary text sequence according to the cross-entropy loss to obtain the inspection report of the hidden defect.

[0139] Specifically, based on the natural language model, the preliminary text sequence is adjusted according to the cross-entropy loss using a rule engine and a language model to obtain a complete inspection report of the hidden defect. Illustratively, the specific probability can be replaced by a textual description according to the needs of the user for digital display, such as replacing "probability 0.82" with "extremely high probability".

[0140] The embodiment calculates the contribution of the intelligent device operating parameters to the probability of causing the hidden defect by Shapley value, and forms a key factor sorting list according to the contribution and the corresponding intelligent device operating parameters; based on the list and the cross-entropy loss function, a natural language model is driven to fill in a predefined template to generate a preliminary text sequence; the logical rationality is checked and the readability is optimized by a rule engine, and a structured inspection report of the hidden defect is output, which visually displays the type of the hidden defect, the reason for causing the hidden defect and the probability, and provides a time-effective reference for intelligent device maintenance and daily use.

[0141] In one embodiment, the visible image data, infrared image data and multi-source sensor data are time-synchronized and feature-fused to obtain a fusion feature vector, including:

[0142] S601, calculate the time difference of the visible image data and the infrared image data, the multi-source sensor data, and the time difference of the infrared image data and the multi-source sensor data, compare the time difference to obtain the maximum time deviation.

[0143] Specifically, the time difference of the visible image data and the infrared image data, the visible image data and the multi-source sensor data, and the infrared image data and the multi-source sensor data is calculated respectively, and the size of the difference is compared to obtain the maximum time deviation.

[0144] S602, calculate the maximum offset according to the maximum time deviation using the following formula:

[0145]

[0146] wherein τ max is the maximum offset, is a function of selecting the maximum value of τ, N is the window length, s recv (n) is the received value of the received signal, s ref (n-τ) is the reference value of the reference signal.

[0147] Specifically, according to the two types of data related to the maximum time deviation, the maximum time deviation of the timestamps of the two types of data is calculated by the cross-correlation function. Illustratively, the maximum offset τ max is the maximum time deviation between the received signal and the reference signal, used to represent the maximum delay between the visible image data and the infrared image data, and the multi-source sensing data. Optionally, the function of selecting the maximum value of τ is to select the maximum value among the time deviation values. Illustratively, the window length N is the cross-correlation calculation window length, used to balance the accuracy and real-time performance, which can be set according to the user's requirements for time alignment, or can be set to 256 by default. Optionally, the received value s recv (n) of the received signal is the received signal of the two types of data related to the maximum time deviation, such as the original sampling value of the vibration sensor in the multi-source sensing data. Illustratively, the reference value s ref (n-τ) of the reference signal is the reference signal of the two types of data related to the maximum time deviation, used for timing calibration, and the frequency needs to be higher than the sensor bandwidth, which can be set according to the actual hardware standard.

[0148] S603, according to the maximum offset, using the following formula, calculate the corrected timestamps of the visible image data, infrared image data and multi-source sensing data:

[0149] δt = τ max · T s

[0150] wherein δt is the corrected timestamp of the data, τ max is the maximum offset, T s is the sampling period of the data.

[0151] Specifically, according to the maximum offset, according to the maximum offset, the original time label of the visible image data, infrared image data and multi-source sensing data is offset, aligned to a unified time reference to obtain, and the corresponding corrected timestamp data is obtained. Illustratively, the corrected timestamp δt of the data is the time label data of the visible image data, infrared image data and multi-source sensing data after calibration. Optionally, the maximum offset τ max is the maximum offset obtained in S602, used to represent the maximum delay between the visible image data and the infrared image data, and the multi-source sensing data. Illustratively, the sampling period T sis the original time tag data of the visible image data, the infrared image data and the multi-source sensing data, and is the sampling period of the output.

[0152] S604, the corrected time stamp is combined with the visible image data, the infrared image data and the multi-source sensing data to obtain corresponding synchronous visible image data, synchronous infrared image data and synchronous multi-source sensing data.

[0153] Specifically, the corrected time stamp of the obtained visible image data, infrared image data and multi-source sensing data is combined with the sampling data of the visible image data, infrared image data and multi-source sensing data to obtain corresponding synchronous visible image data, synchronous infrared image data and synchronous multi-source sensing data.

[0154] S605, if the time interpolation of the synchronous multi-source sensing data is greater than a preset threshold, an interpolation algorithm is used to complete the multi-source sensing data through the following formula to obtain updated synchronous multi-source sensing data:

[0155]

[0156] Wherein, t is the time stamp of data loss, S(t) is the completed data at time stamp t, α is an adjustment parameter, S(t2) is the data at the next sampling time stamp, S(t1) is the data at the previous sampling time stamp, t2 is the next sampling time stamp, t1 is the previous sampling time stamp, S l (t) is a linear completion term, S m (t) is a model compensation term, K is the number of physical model terms, A k is the data transmission amplitude, β is the attenuation coefficient, f k is the dominant frequency of the data signal, φ k is the data transmission phase.

[0157] Specifically, when the time interpolation of the synchronized multi-source sensing data is greater than a preset threshold, an improved interpolation algorithm is adopted, a linear interpolation algorithm and an adaptive weight hybrid model are combined, and updated synchronized multi-source sensing data is obtained. Illustratively, the time interpolation preset threshold can be set according to the user's requirement for the timeliness of the data. Optionally, the timestamp t of the missing data is a detected missing timestamp. Illustratively, the timestamp t completed data S(t) is the completed synchronized multi-source sensing data. Optionally, the adjustment parameter a is used to control the proportion of the model prediction term and the linear interpolation term, which can be optimized through historical data cross-validation. Illustratively, the next sampling timestamp data S(t2) is the sampling data of the next timestamp of the timestamp t of the missing data. Optionally, the previous sampling timestamp data S(t1) is the sampling data of the previous timestamp of the timestamp t of the missing data. Illustratively, the next sampling timestamp t2 is the next sampling timestamp of the timestamp t of the missing data. Optionally, the previous sampling timestamp t1 is the previous sampling timestamp of the timestamp t of the missing data. Illustratively, the linear completion term S l (t) is data obtained according to the linear interpolation algorithm, used to fill in short-term missing data. Optionally, the model compensation term S m (t) is a missing value predicted based on an autoregressive model, used to complete long-term missing or periodic signals. Illustratively, the number of physical model terms K is the model order. Optionally, the data transmission amplitude A k is. Illustratively, the attenuation coefficient β is. Optionally, the dominant frequency f k of the data signal is the dominant frequency of the completed sensing data signal. Illustratively, the data transmission phase φ k is the phase of the sensing data signal.

[0158] S606, morphological distribution features of the synchronized infrared image data are extracted to obtain synchronized infrared feature data.

[0159] Specifically, the contrast, energy and maximum temperature difference features of the synchronized infrared image data are extracted to obtain synchronized infrared feature data. Illustratively, the contrast, energy and other features can be extracted through a gray level co-occurrence matrix.

[0160] S607, the synchronized visible image data, the synchronized infrared feature data and the synchronized multi-source sensing data are fused to obtain a fusion feature vector.

[0161] Specifically, the robot fuses the synchronized visible image data, the synchronized infrared feature data and the synchronized multi-source sensing data to obtain a fusion feature vector. Illustratively, the contrast of the synchronized visible image data is highlighted, the data highlighting the contrast is fused with the synchronized infrared feature data and the synchronized multi-source sensing data to obtain a fusion feature vector.

[0162] The embodiment calculates the maximum time deviation of the visible image data, the infrared feature data and the multi-source sensing data by a cross-correlation function, corrects the time stamp based on a sampling period, obtains corresponding synchronous visible image data, synchronous infrared image data and synchronous multi-source sensing data, adopts adaptive mixed interpolation to complete the numerical value for the missing data segment, obtains updated synchronous multi-source sensing data, and performs feature fusion on the synchronous visible image data, the synchronous infrared feature data and the synchronous multi-source sensing data to obtain a fusion feature vector, thereby providing time-synchronous accurate feature data for hidden defect analysis.

[0163] In one of the embodiments, the feature fusion is performed on the synchronous visible image data, the synchronous infrared feature data and the synchronous multi-source sensing data to obtain a fusion feature vector, including:

[0164] S701, dividing the synchronous visible image data according to a preset segmentation region size to obtain a visible image block set.

[0165] Specifically, the synchronous visible image data is divided according to a preset segmentation region size to obtain a visible image block set. Illustratively, the size of the preset segmentation region is 8x8, which can also be set according to the user's requirement for the visible image segmentation.

[0166] S702, calculating the gray value pixel data of the visible image block set by using the following formula:

[0167]

[0168] wherein h(k) is the number of pixels with a gray value of k, Tile(m, n) is the preset segmentation region, δ is a statistical function, and I(x, y) is the gray value of the coordinate (x, y) in the synchronous visible image data.

[0169] Specifically, the number of pixels with different gray values of the block set is counted according to the obtained visible image block set to obtain the gray value pixel data. Illustratively, the number of pixels with a gray value of k h(k) is the number of pixels with a gray value of k in any image block. Optionally, the preset segmentation region Tile(m, n) is the size of the block region. The statistical function δ is used to record the number of pixels with a gray value of k. Optionally, the gray value I(x, y) of the coordinate (x, y) in the synchronous visible image data is the gray value data of any pixel coordinate (x, y) in any image block.

[0170] S703, combining the gray value pixel data to obtain an initial gray histogram set.

[0171] Specifically, the gray value pixel data of all the image blocks obtained in S702 is combined to obtain an initial gray histogram set.

[0172] S704, limit the initial gray histogram set according to the preset gray value quantity threshold to obtain a cropped gray histogram set, and use the following formula to map the cropped gray histogram set to obtain new gray values:

[0173]

[0174] wherein k ′ is the new gray value obtained by mapping the gray value k in the initial gray histogram set, f is the rounding function, K is the number of gray levels in the initial gray histogram set, h ′ (k) is the number of pixels of the gray value k in the cropped gray histogram set.

[0175] Specifically, according to the preset gray value quantity threshold, the initial gray histogram set is limited to obtain a cropped gray histogram set, and according to the mapping formula and the cropped gray histogram set, the feature range is expanded to enhance the contrast to obtain new gray values. Illustratively, the gray value quantity threshold can be set according to the user's need for image contrast. Optionally, the new gray value k ′ obtained by mapping the gray value k in the initial gray histogram set is the mapped new gray value. Illustratively, the rounding function f is used to round the input object to an integer. Optionally, the number of gray levels K in the initial gray histogram set is the gray value range of the initial gray histogram set obtained in S703. Illustratively, the number of pixels h ′ (k) of the gray value k in the cropped gray histogram set is the number of pixels of the gray value k in the cropped gray histogram set obtained by cropping according to the preset gray value quantity threshold.

[0176] S705, according to the new gray value, mapping the initial gray histogram set to obtain a new gray histogram set.

[0177] Specifically, the new gray values obtained in S706 are used to replace the original gray values in the initial gray histogram set to obtain a new gray histogram set.

[0178] S706, using the following formula, bilinear interpolation is performed on the new gray histogram set to obtain enhanced synchronous image data:

[0179]

[0180] wherein I c is the enhanced synchronous image data, w ij is the adjacent weight of the coordinate (i, j), is the new gray histogram set of the segmented region with the coordinate (i, j), and I(i, j) is the gray value of the coordinate (i, j) in the synchronous visible image data.

[0181] Specifically, the new gray histogram set is bilinearly interpolated according to the formula, the image is restored, and enhanced synchronous image data is obtained. Illustratively, the enhanced synchronous image data I c is pixel-enhanced image data. Optionally, the adjacent weight w ij is used for distance-proportional distribution. Illustratively, the new gray histogram set of the segmented region with coordinates (i, j) is the new gray histogram set of the image patch with center coordinates (i, j). Optionally, the gray value I(i, j) in the synchronous visible image data with coordinates (i, j) is the gray value in the synchronous visible image data input by S701 with coordinates (i, j).

[0182] S707, principal component feature extraction and feature dimension reduction processing are performed on the enhanced synchronous image data by using principal component analysis, and reserved feature image data is obtained.

[0183] Specifically, the first n principal components are reserved by using principal component analysis, and the reserved feature image data after dimension reduction is obtained. Illustratively, n can be set according to the user's demand for gray feature extraction.

[0184] S708, the reserved feature image data, the synchronous infrared feature data, and the synchronous multi-source sensing data are weighted and fused by using an attention mechanism network, and a fusion feature vector is obtained.

[0185] Specifically, the reserved feature image data, the synchronous infrared feature data, and the synchronous multi-source sensing data are weighted and fused by using an attention mechanism network, and a fusion feature vector is obtained.

[0186] This embodiment obtains, by local enhancement, feature compression, and cross-modal attention, to provide high-precision and low-latency multi-modal data for hidden defect class extraction and reason inference.

[0187] In one of the embodiments, based on a convolutional neural network, hidden defects of the fusion feature vector are extracted, and hidden defect classes are obtained, including:

[0188] S801, the fusion feature vector is converted into a two-dimensional structure, and a two-dimensional fusion feature map is obtained.

[0189] Specifically, the robot converts the one-dimensional feature vector into a spatially interpretable two-dimensional structure to obtain a two-dimensional fusion feature map to adapt to the local feature extraction requirements of the convolutional neural network. Illustratively, the height and width of the two-dimensional fusion feature map can be set according to the user's feature extraction budget.

[0190] S802, based on a convolutional neural network, the two-dimensional fusion feature map is input into a defect classification model, and hidden classification classes are obtained.

[0191] wherein the defect classification model comprises:

[0192] an input layer 8021 configured to input the two-dimensional fusion feature map.

[0193] Specifically, the two-dimensional fusion feature map obtained by the inputting S801 is inputted.

[0194] a multi-scale convolution layer 8022 configured to extract local features and intermediate features of the two-dimensional fusion feature map to obtain a defect feature map, and an output of each layer is taken as an input of a next layer; a convolution output formula in the multi-scale convolution layer is: wherein is an output of the lth multi-scale convolution layer, ReLU is an activation function, is a convolution kernel of the lth multi-scale convolution layer, is an output of an (l-1)th multi-scale convolution layer, b (l) is a bias term of the lth multi-scale convolution layer.

[0195] Specifically, the multi-scale convolution layer is adopted to extract local and global features from the two-dimensional feature map to output the defect feature map. Illustratively, a number of layers of the multi-scale convolution layer can be set according to a precision and a budget of feature extraction of a user. Illustratively, the output of the lth multi-scale convolution layer is is an output of the lth layer in the multi-scale convolution layer. Illustratively, the activation function ReLU is used to introduce nonlinearity and accelerate convergence. Illustratively, the convolution kernel of the lth multi-scale convolution layer can be set according to a feature of a feature to be extracted and an input matrix. Illustratively, the output of the (l-1)th multi-scale convolution layer is is the (l-1)th layer in the multi-scale convolution layer. Illustratively, the bias term b (l) of the lth multi-scale convolution layer can be set according to an output channel of the convolution kernel.

[0196] a pooling layer 8023 configured to compress the defect feature map to obtain a compressed defect feature map; an output formula of the pooling layer is: wherein P i,j is the compressed defect feature map, is a maximum value operation, is the output of the multi-scale convolution layer.

[0197] Specifically, the compressed defect feature map is obtained by compressing the defect feature map to reduce computational complexity and enhance feature translation invariance. Illustratively, the compressed defect feature map P i,j is the defect feature map compressed and reduced in dimension. Illustratively, the maximum value operation is used to retain significant features and reduce dimensions and suppress noise. Illustratively, the output of the multi-scale convolution layer is is the output of the last multi-scale convolution layer.

[0198] A full connection layer 8024 is configured to determine a defect category of the compressed defect feature map, and obtain a hidden defect category.

[0199] Specifically, the compressed defect feature map after the pooling is flattened into a one-dimensional vector, and is mapped to a defect category space through the full connection layer to calculate a defect category probability distribution, and obtain the most possible hidden defect category.

[0200] An output layer 8025 is configured to output the hidden defect category.

[0201] Specifically, the hidden defect category obtained according to the fusion feature vector analysis is output.

[0202] The embodiment converts the fusion feature vector into a two-dimensional structure, adapts the defect features of different scales through a mixed convolution kernel size, and obtains a high-precision hidden defect category, thereby providing an accurate defect category for reasoning of a hidden defect reason.

[0203] In the above-mentioned intelligent device inspection method of fusing robot vision, the visible image data, the infrared image data and the multi-source sensing data of the intelligent device are acquired, the hidden defect category is recognized through multi-modal fusion, and the reason and the probability of causing the defect are reasoned based on the Bayesian network algorithm according to the hidden defect category, so as to realize the defect type obtained through the defect detail feature and the reason of forming the defect obtained through the reverse deduction, and provide a time-effective suggestion for product maintenance management decision.

[0204] It should be understood that, although each step in the flowchart involved in each embodiment as described above is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps.

[0205] Based on the same inventive concept, the embodiments of the present application also provide a device for implementing the intelligent device inspection method involving robot vision. The device provides a solution similar to the implementation described in the above method, so the specific limitations in one or more device embodiments involving robot vision provided below can refer to the limitations of the intelligent device inspection method involving robot vision described above, and will not be repeated here.

[0206] In one exemplary embodiment, as shown in Figure 2 A device for inspecting intelligent devices involving robot vision 200 is provided, comprising:

[0207] An information acquisition module 201 is configured to acquire visible image data, infrared image data, and multi-source sensing data of the intelligent device;

[0208] An information fusion module 202 is configured to time synchronize and feature fuse the visible image data, the infrared image data, and the multi-source sensing data to obtain a fusion feature vector;

[0209] A defect extraction module 203 is configured to extract hidden defects of the fusion feature vector based on a convolutional neural network to obtain a hidden defect category;

[0210] A report generation module 204 is configured to obtain an inspection report of the hidden defects based on a Bayesian network algorithm according to the hidden defect category; the inspection report includes the hidden defect category and a defect cause.

[0211] Further, the report generation module further comprises:

[0212] A cause-effect generation unit is configured to match a preset defect element knowledge base according to the hidden defect category to obtain a dynamic defect cause-effect diagram;

[0213] A probability calculation unit is configured to calculate a defect root cause probability distribution in the dynamic defect cause-effect diagram based on a Bayesian network algorithm;

[0214] A report generation unit is configured to convert the defect root cause probability distribution into a text report based on a natural language model to obtain the inspection report of the hidden defects.

[0215] Further, the cause-effect generation unit is further configured to:

[0216] Match the defect element knowledge base according to the hidden defect category to obtain corresponding defect cause data using the following formula:

[0217]

[0218] Wherein, Rule(Cd ) is defect cause data, X i is the i-th defect cause in defect meta-knowledge base, w i is the causal weight of the i-th defect cause, C d is a hidden defect category, is a set of defect causes in defect meta-knowledge base;

[0219] The defect cause data is expanded to obtain dynamic defect causal data by using the following formula combined with the operating parameters of the intelligent device:

[0220] G(V, E) = Rule(C d )∪(p j →X i )

[0221] Wherein, G(V, E) is dynamic defect causal data, p j is the j-th operating parameter of the intelligent device, X i is the i-th defect cause in defect meta-knowledge base;

[0222] The dynamic defect causal data is combined to obtain a dynamic defect causal graph.

[0223] Further, the probability calculation unit is also used for:

[0224] The probability of the cause of the defect in the dynamic defect causal graph is calculated to obtain a defect root cause probability by using the following formula:

[0225]

[0226] Wherein, P(X i =1|Pa(X i )) is a defect root cause probability, Pa(X i ) is a set of operating parameters of the intelligent device in the dynamic defect causal data, σ is an activation function, β j is a regression coefficient, p j is an influence coefficient of the j-th operating parameter of the intelligent device, and γ is a bias term.

[0227] The defect root cause probability is iteratively sampled for a preset number of times to obtain a defect root cause probability distribution.

[0228] Further, the report generation unit is also used for:

[0229] The probability distribution contribution in the defect root cause probability distribution is calculated based on the extension of the Shapley value in game theory by using the following formula:

[0230]

[0231] Wherein, φ jis the probability distribution contribution of the jth intelligent device operating parameter, S is a subset of intelligent device operating parameters contained in the defect root cause probability distribution, N is a set of intelligent device operating parameters, f(S) is a defect root cause probability distribution of the subset of intelligent device operating parameters;

[0232] combining the probability distribution contribution and the intelligent device operating parameter corresponding to the probability distribution contribution, a key factor list is obtained;

[0233] traversing the key factor list and the hidden defect category, a preliminary text sequence is generated according to a preset sorting rule;

[0234] Using the following formula, the cross-entropy loss is calculated according to the key factor list:

[0235]

[0236] wherein, is the cross-entropy loss, T is the total number of words in the preliminary text sequence, w t The tth word of the preliminary text sequence, is the key factor list;

[0237] Based on the natural language model, the preliminary text sequence is adjusted according to the cross-entropy loss to obtain the inspection report of the hidden defect.

[0238] Further, the information fusion module further comprises:

[0239] A maximum time difference calculation unit is configured to calculate the time difference values of the visible image data and the infrared image data, the multi-source sensing data, and the time difference values of the infrared image data and the multi-source sensing data, compare the time difference values, and obtain the maximum time deviation.

[0240] A maximum offset calculation unit is configured to calculate the maximum offset according to the maximum time deviation using the following formula:

[0241]

[0242] wherein, max is the maximum offset, is a function of selecting the maximum value of tau, N is the window length, s recv (n) is the received value of the received signal, s ref (n-tau) is the reference value of the reference signal.

[0243] A correction time unit is configured to calculate the correction time stamp of the visible image data, the infrared image data and the multi-source sensing data according to the maximum offset using the following formula:

[0244] delta t = tau max · T s

[0245] wherein δt is the corrected timestamp of the data, τ max is the maximum offset, T s is the sampling period of the data;

[0246] a time data combination unit, configured to combine the corrected timestamp with the visible image data, the infrared image data and the multi-source sensing data to obtain corresponding synchronous visible image data, synchronous infrared image data and synchronous multi-source sensing data;

[0247] an interpolation completion unit, configured to, if the time interpolation of the synchronous multi-source sensing data is greater than a preset threshold, complete the multi-source sensing data by using an interpolation algorithm through the following formula to obtain updated synchronous multi-source sensing data:

[0248]

[0249] wherein t is the timestamp of the missing data, S(t) is the completed data at the timestamp t, a is an adjustment parameter, S(t2) is the data at the next sampling timestamp, S(t1) is the data at the previous sampling timestamp, t2 is the next sampling timestamp, t1 is the previous sampling timestamp, S l (t) is a linear completion term, S m (t) is a model compensation term, K is the number of physical model terms, A k is the data transmission amplitude, β is an attenuation coefficient, f k is the dominant frequency of the data signal, φ k is the data transmission phase;

[0250] an infrared feature extraction unit, configured to extract the morphological distribution features of the synchronous infrared image data to obtain synchronous infrared feature data;

[0251] a feature fusion unit, configured to perform feature fusion on the synchronous visible image data, the synchronous infrared feature data and the synchronous multi-source sensing data to obtain a fusion feature vector.

[0252] Further, the feature fusion unit is further configured to:

[0253] divide the synchronous visible image data according to a preset segmentation region size to obtain a visible image block set;

[0254] calculate the gray value pixel data of the visible image block set by using the following formula:

[0255]

[0256] wherein h(k) is the number of pixels with a gray value of k, Tile(m, n) is the preset segmentation region, δ is a statistical function, and I(x, y) is the gray value of the coordinate (x, y) in the synchronous visible image data;

[0257] combining the gray value pixel data to obtain an initial gray histogram set;

[0258] According to a preset gray value quantity threshold, the initial gray histogram set is limited to obtain a cropped gray histogram set, and the cropped gray histogram set is mapped to obtain new gray values using the following formula:

[0259]

[0260] wherein, k ′ is the new gray value obtained by mapping the gray value k in the initial gray histogram set, f is an integer function, K is the number of gray levels in the initial gray histogram set, h ′ (k) is the number of pixels of the gray value k in the cropped gray histogram set;

[0261] According to the new gray values, the initial gray histogram set is mapped to obtain a new gray histogram set;

[0262] The new gray histogram set is bilinearly interpolated using the following formula to obtain enhanced synchronous image data:

[0263]

[0264] wherein, I c is the enhanced synchronous image data, w ij is the adjacent weight of the coordinate (i, j), is the new gray histogram set of the segmented region of the coordinate (i, j), and I(i, j) is the gray value of the coordinate (i, j) in the synchronous visible image data.

[0265] The principal component analysis method is used to extract principal component features and reduce feature dimensions of the enhanced synchronous image data to obtain a reserved feature image data.

[0266] The attention mechanism network is used to weight and fuse the reserved feature image data, the synchronous infrared feature data and the synchronous multi-source sensing data to obtain a fusion feature vector.

[0267] Further, the defect extraction module is further used for:

[0268] The fusion feature vector is converted into a two-dimensional structure to obtain a two-dimensional fusion feature map.

[0269] The two-dimensional fusion feature map is input into a defect classification model based on a convolutional neural network to obtain a hidden classification category.

[0270] The defect classification model comprises:

[0271] An input layer for inputting the two-dimensional fusion feature map.

[0272] a plurality of multi-scale convolution layers, used for extracting local features and intermediate features of the two-dimensional fusion feature map to obtain a defect feature map, an output of each layer being used as an input of a next layer; a convolution output formula in the multi-scale convolution layer being: wherein is an output of the lth multi-scale convolution layer, ReLU is an activation function, is a convolution kernel of the lth multi-scale convolution layer, is an output of an (l-1)th multi-scale convolution layer, b (l) is a bias term of the lth multi-scale convolution layer;

[0273] a pooling layer, used for compressing the defect feature map to obtain a compressed defect feature map; an output formula of the pooling layer being: wherein, P i,j is the compressed defect feature map, is a maximum value operation, is the output of the multi-scale convolution layer;

[0274] a full connection layer, used for determining a defect category of the compressed defect feature map to obtain a hidden defect category;

[0275] an output layer, used for outputting the hidden defect category.

[0276] In an embodiment, a robot is provided, comprising a memory, a processor and a sensor, the memory storing a computer program, characterized in that the processor implements a method for intelligent device inspection of fusion robot vision as described above when executing the computer program.

[0277] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts are described in the part of the method embodiment. The device embodiments described above are merely illustrative, wherein the components described as separate components can be or can not be physically separated, and the components displayed as units can be or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present disclosure according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0278] The above-described embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the patent scope of the application. It should be noted that, for those skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which are all within the protection scope of the present application.

Claims

1. A method for intelligent device inspection with fusion of robot vision, characterized in that, The method comprises: acquiring visible image data, infrared image data and multi-source sensing data of an intelligent device; time synchronizing and feature fusing the visible image data, the infrared image data and the multi-source sensing data to obtain a fusion feature vector; based on a convolutional neural network, extracting hidden defects of the fusion feature vector to obtain a hidden defect category; based on a Bayesian network algorithm, obtaining an inspection report of the hidden defects according to the hidden defect category; the inspection report comprises the hidden defect category and a defect reason; wherein, based on the Bayesian network algorithm, the inspection report of the hidden defects is obtained according to the hidden defect category; the inspection report comprises the hidden defect category and a defect reason, which comprises: matching a preset defect element knowledge base according to the hidden defect category to obtain a dynamic defect causal graph; based on a Bayesian network algorithm, calculating a reason probability of causing defects in the dynamic defect causal graph to obtain a defect root cause probability distribution; based on a natural language model, converting the defect root cause probability distribution into a text report to obtain the inspection report of the hidden defects; wherein, based on the natural language model, the inspection report of the hidden defects is obtained by converting the defect root cause probability distribution into a text report, which comprises: using the following formula, calculating the probability distribution contribution in the defect root cause probability distribution based on the extension of the Shapley value in game theory: ； wherein, is the probability distribution contribution of the th smart device operating parameter, is a subset of smart device operating parameters contained in the defect root cause probability distribution, is a set of smart device operating parameters, is a defect root cause probability distribution for a subset of smart device operating parameters; combining the probability distribution contribution and the intelligent device operation parameter corresponding to the probability distribution contribution to obtain a key factor list; traversing the key factor list and the hidden defect category, and generating a preliminary text series according to a preset sorting rule; using the following formula, calculating the cross-entropy loss according to the key factor list: ； in, It is cross-entropy loss. This is the total number of words in the initial text sequence. The first of the preliminary text sequence One word, It is a list of key factors; based on a natural language model, adjusting the preliminary text series according to the cross-entropy loss to obtain the inspection report of the hidden defects.

2. The method of claim 1, wherein, According to the hidden defect category, the dynamic defect causal graph is obtained by matching a preset defect element knowledge base, which comprises: using the following formula, matching the defect element knowledge base according to the hidden defect category to obtain corresponding defect reason data: ； wherein, is defect cause data, is a th defect cause in the defect meta-knowledge base, is a th defect cause, is a causal weight of the th defect cause, is a hidden defect category, is a set of defect causes in the defect meta-knowledge base; using the following formula, expanding the defect reason data in combination with the operation parameters of the intelligent device to obtain dynamic defect causal data: ； wherein, is dynamic defect causal data, is intelligent device first item operating parameter, is defect cause in the defect meta-knowledge base, th defect cause; the dynamic defect causal data is combined to obtain the dynamic defect causal graph.

3. The method of claim 1, wherein, Based on the Bayesian network algorithm, the defect root cause probability distribution is obtained by calculating the reason probability of causing defects in the dynamic defect causal graph, which comprises: using the following formula, calculating the reason probability of causing defects in the dynamic defect causal graph to obtain a defect root cause probability: ； in, It is the probability of the root cause of the defect. It is a set of intelligent device operating parameters in dynamic defect causal data. It is an activation function. It is the regression coefficient. It is the first smart device The influence coefficient of the operating parameters It is a bias term; iterative sampling of the defect root cause probability is performed for a preset number of times to obtain the defect root cause probability distribution.

4. The method of claim 1, wherein, The visible image data, the infrared image data and the multi-source sensing data are time synchronized and feature fused to obtain a fusion feature vector, which comprises: calculating the time difference of the visible image data, the infrared image data and the multi-source sensing data, and the time difference of the infrared image data and the multi-source sensing data, comparing the time difference to obtain a maximum time deviation; Using the following formula, the maximum offset is calculated according to the maximum time deviation: ； wherein is a maximum offset, is a function of selecting a maximum value, is a window length, is a reception value of the reception signal, is a reference value of the reference signal; According to the maximum offset, the corrected timestamps of the visible image data, the infrared image data and the multi-source sensing data are calculated using the following formula: ； wherein, is a modified timestamp of the data, is a maximum offset, is a sampling period of the data; The corrected timestamps are combined with the visible image data, the infrared image data and the multi-source sensing data to obtain corresponding synchronized visible image data, synchronized infrared image data and synchronized multi-source sensing data; If the time interpolation of the synchronized multi-source sensing data is greater than a preset threshold, an interpolation algorithm is used to complete the multi-source sensing data through the following formula to obtain updated synchronized multi-source sensing data: ； ； wherein is the time stamp of the data missing, is the time stamp is the completed data, is the adjustment parameter, is the data of the next sampling time stamp, is the data of the previous sampling time stamp, is the next sampling time stamp, is the previous sampling time stamp, is the linear completion term, is the model compensation term, is the number of physical model terms, is the data transmission amplitude, is the attenuation coefficient, is the data signal dominant frequency, is the data transmission phase; The morphological distribution features of the synchronized infrared image data are extracted to obtain synchronized infrared feature data; The synchronized visible image data, the synchronized infrared feature data and the synchronized multi-source sensing data are feature fused to obtain the fusion feature vector.

5. The method of claim 4, wherein, The synchronized visible image data, the synchronized infrared feature data and the synchronized multi-source sensing data are feature fused to obtain the fusion feature vector, including: The synchronized visible image data is segmented according to a preset segmentation region size to obtain a visible image block set; The gray value pixel data of the visible image block set is calculated using the following formula: ； wherein, is the number of pixels with a gray value of is a preset segmentation region, is a statistical function, is a gray value in the synchronized visible image data at the coordinate . The gray value pixel data is combined to obtain an initial gray histogram set; According to a preset gray value quantity threshold, the initial gray histogram set is limited to obtain a cropped gray histogram set, and the cropped gray histogram set is mapped to obtain new gray values using the following formula: ； wherein, is the initial gray value of the gray value mapped to the new gray value, is the rounding function, is the number of gray levels in the initial gray histogram set, is the number of pixels of the gray value in the cropped gray histogram set; According to the new gray values, the initial gray histogram set is mapped to obtain a new gray histogram set; The new gray histogram set is bilinearly interpolated using the following formula to obtain enhanced synchronized image data: ； wherein, is enhanced synchronized image data, is a neighboring weight with coordinates is a new histogram set of the segmented region with coordinates is a gray value in the synchronized visible image data with coordinates ;​​ Principal component analysis is used to extract principal component features and perform feature dimension reduction processing on the enhanced synchronized image data to obtain reserved feature image data; The reserved feature image data, the synchronized infrared feature data and the synchronized multi-source sensing data are weighted fused through an attention mechanism network to obtain the fusion feature vector.

6. The method of claim 1, wherein, The fusion feature vector is converted into a two-dimensional structure to obtain a two-dimensional fusion feature map; The two-dimensional fusion feature map is input into a defect classification model based on a convolutional neural network to obtain the hidden classification category; The defect classification model includes: An input layer for inputting the two-dimensional fusion feature map; A fully connected layer for determining the defect category of the compressed defect feature map to obtain the hidden defect category; a plurality of multi-scale convolutional layers for extracting local features and intermediate features of the two-dimensional fusion feature map to obtain a defect feature map, an output of each layer being used as an input of a next layer; a convolution output formula in the plurality of multi-scale convolutional layers being: wherein is an output of an i-th layer multi-scale convolutional layer, is an activation function, is a convolution kernel of the i-th layer multi-scale convolutional layer, is an output of an i-th layer multi-scale convolutional layer, is a bias term of the i-th layer multi-scale convolutional layer. a pooling layer, configured to compress the defect feature map to obtain a compressed defect feature map; an output formula of the pooling layer is: wherein, is the compressed defect feature map, is a maximum value taking operation, is an output of the multi-scale convolution layer; An output layer for outputting the hidden defect category. The device includes:

7. An intelligent device inspection device fusing robot vision, characterized in that, An information acquisition module for acquiring visible image data, infrared image data and multi-source sensing data of a smart device; An information fusion module for time synchronizing and feature fusing the visible image data, the infrared image data and the multi-source sensing data to obtain a fusion feature vector; ​ The defect extraction module is configured to extract hidden defects of the fusion feature vector based on a convolutional neural network to obtain a hidden defect category. The report generation module is configured to obtain an inspection report of the hidden defect based on a Bayesian network algorithm according to the hidden defect category; the inspection report includes the hidden defect category and a defect cause. The report generation module further includes: The cause-effect generation unit is configured to match a preset defect meta-knowledge base according to the hidden defect category to obtain a dynamic defect cause-effect diagram and a set of intelligent device operation parameters. The probability calculation unit is configured to calculate a probability of a cause of the defect in the dynamic defect cause-effect diagram based on a Bayesian network algorithm to obtain a defect root cause probability distribution. The report generation unit is configured to convert the defect root cause probability distribution into a text report based on a natural language model to obtain the inspection report of the hidden defect. The report generation unit is further configured to: calculate a probability distribution contribution in the defect root cause probability distribution based on an extension of a Shapley value in game theory using the following formula: ； wherein, is a probability distribution contribution of the th smart device operating parameter, is a subset of smart device operating parameters contained in the defect root cause probability distribution, is a set of smart device operating parameters, is a defect root cause probability distribution of the subset of smart device operating parameters; combine the probability distribution contribution and an intelligent device operation parameter corresponding to the probability distribution contribution to obtain a key factor list; traverse the key factor list and the hidden defect category, and generate a preliminary text series according to a preset sorting rule; calculate a cross-entropy loss according to the key factor list using the following formula: ； in, It is cross-entropy loss. This is the total number of words in the initial text sequence. The first of the preliminary text sequence One word, It is a list of key factors; adjust the preliminary text series according to the cross-entropy loss based on the natural language model to obtain the inspection report of the hidden defect.

8. A robot comprising a memory, a processor and a sensor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 6. The processor executes the computer program to implement the steps of the method in any one of claims 1 to 6.

Images