Intelligent Image Data Recognition and Analysis Method, System and Device Based on AI Model

Abstract

The present invention discloses an intelligent image data recognition and analysis method, system and device based on an AI model, belonging to the technical field of image data analysis. The present invention obtains image data, sound data, physical parameter data and environmental data during the welding process of a welding robot, and integrates them to form real-time multimodal welding data; uses historical data trained by a deep neural network for anomaly classification; inputs sound data and physical parameter data into the deep neural network to obtain an abnormal classification set of welding data; extracts image data features, sets a normal feature range, compares the image data features with the normal feature range to obtain abnormal multimodal welding data; uses a weighted directed graph to determine the environment-welding association relationship, specifically the association relationship between different factors in the environmental data and different abnormal categories; based on the environment-welding association relationship, excludes the anomalies caused by environmental factors in the abnormal multimodal welding data.

Description

Technical Field

[0001] The present invention relates to the technical field of image data analysis, specifically an intelligent image data recognition and analysis method, system and device based on an AI model. Background Art

[0002] In modern industrial production, welding, as a key joining process, is widely used in many fields such as automobile manufacturing, aerospace, and shipbuilding. With the development of industrial automation and intelligence, the requirements for welding quality and production efficiency are increasing day by day.

[0003] Existing abnormal judgment means are mostly based on simple threshold settings. When the monitored data exceeds or is lower than the set threshold, it is determined as abnormal, which is very prone to false positives and false negatives and cannot adapt to diverse welding working conditions. The welding process is affected by multiple factors, but the existing technology lacks in-depth analysis of the correlation between these factors and cannot accurately judge the root cause of abnormal occurrences. Although some monitoring systems have introduced image data, the processing and analysis of image data are relatively elementary and can only perform simple image recognition. Summary of the Invention

[0004] The purpose of the present invention is to provide an intelligent image data recognition and analysis method, system and device based on an AI model to solve the problems raised in the existing technology.

[0005] To achieve the above purpose, the present invention provides the following technical solutions:

[0006] In the first aspect, the present invention provides an intelligent image data recognition and analysis method based on an AI model, including the following steps:

[0007] Obtain image data, sound data, physical parameter data, and environmental data during the welding process of a welding robot, perform time synchronization and data cleaning, and integrate them to form real-time multimodal welding data;

[0008] Collect historical sound data and historical physical parameter data, use a deep neural network for training, and perform abnormal classification; based on the real-time multimodal welding data, input the sound data and physical parameter data into the deep neural network to obtain a welding data abnormal classification set;

[0009] Extract image data features, including overall welding area features, weld close-up features, and molten pool features; set a normal feature range, compare the image data features with the normal feature range to obtain abnormal multimodal welding data;

[0010] Based on the abnormal classification set of welding data, combined with the corresponding environmental data, a weighted directed graph is used to determine the environment-welding association relationship, specifically the association relationship between different factors in the environmental data and different abnormal categories; based on the environment-welding association relationship, the abnormalities caused by environmental factors in the abnormal multimodal welding data are excluded.

[0011] Combined with the first aspect, in the first implementation manner of the first aspect of the present application, the method for obtaining image data, sound data, physical parameter data, and environmental data during the welding process of a welding robot, performing time synchronization and data cleaning, and integrating them to form real-time multimodal welding data includes:

[0012] Using an industrial camera to take pictures of the overall welding area, the weld close-up, and the molten pool to obtain image data; using a microphone to collect the sound signals generated during the welding process to obtain sound data;

[0013] Using a current sensor and a voltage sensor to measure the welding current and voltage respectively, using a speed sensor to monitor the welding speed and wire feeding speed, using a joint angle sensor to obtain the angle information of each joint of the welding robot, using a position sensor to determine the position coordinates of the robot, and using a temperature sensor to measure the temperature of each component of the welding equipment; integrating to obtain physical parameter data;

[0014] Using a temperature sensor and a humidity sensor to obtain the environmental temperature and humidity of the welding area, and using a light sensor to obtain the light intensity of the welding area; integrating to obtain environmental data;

[0015] Performing time synchronization and data cleaning, and integrating to form real-time multimodal welding data.

[0016] Combined with the first aspect, in the second implementation manner of the first aspect of the present application, the method for collecting historical sound data and historical physical parameter data, training using a deep neural network, and performing abnormal classification includes:

[0017] Labeling the historical sound data and historical physical parameter data, and classifying the data into normal samples and abnormal samples; for abnormal samples, further classifying them into different types of abnormalities, including welding equipment-related abnormalities, welding process-related abnormalities, and welding quality-related abnormalities;

[0018] A deep neural network using multimodal fusion includes an input layer, a feature extraction layer, a fusion layer, and a classification layer; in the input layer, a sound data input layer and a physical parameter data input layer are respectively set, the sound data input layer receives the normalized sound feature vector, and the physical parameter data input layer receives the normalized physical parameter data; in the feature extraction layer, for sound data, a convolutional neural network is used for feature extraction, and for physical parameter data, a fully connected layer is used for preliminary feature transformation; in the fusion layer, the sound feature vector extracted by the convolutional neural network and the physical parameter feature vector output by the fully connected layer are concatenated to obtain a fused feature vector; in the classification layer, a fully connected layer and a Softmax activation function are used for anomaly classification;

[0019] Perform data partitioning, train the deep neural network, perform hyperparameter tuning, and perform model evaluation and optimization.

[0020] Combined with the first aspect, in the third implementation manner of the first aspect of the present application, the performing data partitioning, training the deep neural network, performing hyperparameter tuning, and performing model evaluation and optimization includes:

[0021] Fuse the historical sound data and historical physical parameter data, and divide them into a training set, a validation set, and a test set; define a cross-entropy loss function, use Adam as the optimization algorithm to update the parameters of the deep neural network; during training, input the training data into the deep neural network in batches, calculate the value of the loss function, and update the parameters of the deep neural network; use grid search to evaluate the performance under different hyperparameter combinations on the validation set, and select the optimal hyperparameter combination;

[0022] Use the test set to evaluate the trained deep neural network, calculate the classification accuracy, recall rate, and F1 value, and perform optimization of the deep neural network.

[0023] Combined with the first aspect, in the fourth implementation manner of the first aspect of the present application, the inputting sound data and physical parameter data into the deep neural network based on real-time multimodal welding data to obtain a welding data anomaly classification set includes:

[0024] Based on real-time multimodal welding data, input sound data and physical parameter data into the deep neural network, set a threshold according to the probability output by the Softmax function, and the category with a probability value greater than the threshold is determined as the category to which the sample belongs; summarize the anomaly classification results to generate a welding data anomaly classification set, which includes the welding data anomaly situations at different time points.

[0025] Combined with the first aspect, in the fifth implementation manner of the first aspect of the present application, the extracting image data features, including the overall features of the welding area, the close-up features of the weld seam, and the features of the molten pool, includes:

[0026] The method for extracting the overall features of the welding area is as follows: Use the Canny algorithm to identify the workpiece contour, and determine the position of the workpiece in the welding area through contour matching; Use the SIFT algorithm to extract the feature points on the workpiece, and combine the three-dimensional model and the camera calibration parameters to calculate the pose information of the workpiece, including the rotation angle and the translation vector;

[0027] Let the point set on the workpiece contour be Find the position most similar to the standard template contour through the contour matching algorithm to obtain the position coordinates of the workpiece in this coordinate system Here, is the position parameter of the workpiece in the welding area.

[0028] Use the SIFT algorithm to extract the feature points on the workpiece. Let the extracted feature point set be . Combine the three-dimensional model and the camera calibration parameters. The internal parameter matrix of the camera is , where , are the components of the focal length in the x and y directions, , is the image center coordinate. The external parameter matrix is , R is the rotation matrix, and t is the translation vector. Calculate the pose information of the workpiece through feature point matching and the principle of triangulation. The rotation angle θ is calculated from the elements of the rotation matrix R. For example, the rotation angle around the z-axis ; The translation vector is directly given by the external parameter matrix. These parameters accurately describe the pose of the workpiece and provide a basis for judging whether the workpiece has displacement or pose change during the welding process.

[0029] The method for extracting the close-up features of the weld is as follows: Segment and extract the weld area from the background through the region growing algorithm, and calculate the width of the weld at different positions; Use the principle of structured light measurement, combined with the image data, to obtain the height information of the weld; During the welding process, project light with a known structure onto the weld, and by taking pictures of the deformed image of the light on the weld surface, calculate the weld height according to the principle of triangulation; Calculate the root mean square roughness of the weld surface, and through the analysis of the weld surface texture, use the gray-level co-occurrence matrix method to extract the texture features;

[0030] After segmenting and extracting the weld area from the background through the region growing algorithm, let the pixel point set of the weld area in the image be . In the image coordinate system, calculate the width of the weld at different positions. For a certain position i, find two edge points and perpendicular to the weld direction at this position, then the weld width at this position , in pixels, and then converted into the actual weld width dimension according to the conversion relationship between the pixels calibrated by the camera and the actual size.

[0031] Using the principle of structured light measurement, assume that the coordinates of the structured light pattern projected onto the weld surface on the camera imaging plane are . According to the principle of triangulation, given the relative position relationship (baseline distance b) between the structured light projector and the camera and the internal parameter matrix K of the camera, for a certain pixel point , its corresponding three-dimensional space point is calculated by the following formula: , and then combined with the geometric information of the structured light pattern and the known projection angle , through calculate the height of this point, so as to obtain the height information of the weld.

[0032] Calculate the root mean square roughness of the weld surface , assume that the pixel gray value matrix of a certain local area on the weld surface is , the root mean square roughness , where is the average value of the pixel gray levels in this local area. The gray level co-occurrence matrix method is used to extract texture features. Assume that the gray level co-occurrence matrix is , L is the number of gray levels, and by calculating the eigenvalues of the gray level co-occurrence matrix, such as contrast, correlation, energy, entropy, etc., to describe the texture features of the weld surface, and these texture features can be used to judge the quality of the weld and whether there are defects.

[0033] The method for extracting the molten pool features is as follows: Use the morphological processing algorithm to process the molten pool image and outline the molten pool contour; by calculating the perimeter, area and aspect ratio of the molten pool contour, describe the shape features of the molten pool.

[0034] Assume that the point set on the molten pool contour is , calculate the perimeter of the molten pool contour, the area ; the aspect ratio , where and are the length and width of the circumscribed rectangle of the molten pool contour respectively. Through these parameters, the shape features of the molten pool are comprehensively described, providing a quantitative basis for judging whether the state of the molten pool is normal.

[0035] Combining with the first aspect, in the sixth implementation manner of the first aspect of this application, the setting of the normal feature range, comparing the image data features with the normal feature range, and obtaining abnormal multimodal welding data, including:

[0036] Under the normal working condition of the welding robot, extract the mean and standard deviation of the data in the overall characteristics of the welding area, the close-up characteristics of the weld seam, and the characteristics of the molten pool, and set the normal characteristic range; compare the characteristics of the image data with the normal characteristic range, mark the abnormal data, and obtain the abnormal multi-modal welding data.

[0037] Combined with the first aspect, in the seventh implementation manner of the first aspect of the present application, based on the abnormal classification set of welding data, combined with the corresponding environmental data, use a weighted directed graph to determine the environment-welding association relationship, specifically the association relationship between different factors in the environmental data and different abnormal categories, including:

[0038] Create nodes for the environmental temperature, humidity, and light intensity respectively to determine the environmental factor nodes; create nodes for each abnormal category in the abnormal classification set of welding data to determine the abnormal category nodes; for each welding abnormal category, count the corresponding environmental data situation before or when the abnormal occurs; calculate the weight of the edge using conditional probability.

[0039] When there is an association between a certain environmental factor and a certain abnormal category, draw a directed edge from the environmental factor node to the corresponding abnormal category node. Among them, the method for judging whether there is an association is: if the calculated weight is not 1, it indicates that there is a non-random association relationship between the two; determine the environment-welding association relationship.

[0040] Combined with the first aspect, in the eighth implementation manner of the first aspect of the present application, based on the environment-welding association relationship, exclude the abnormalities caused by environmental factors in the abnormal multi-modal welding data, including:

[0041] For the direct connection edges from the environmental factor nodes to the abnormal category nodes in the weighted directed graph, when the weight of a certain edge is greater than 1, it indicates that the environmental factor has a direct promoting effect on the abnormal category, and when the weight of a certain edge is less than 1, it indicates that the environmental factor has a negative direct association with the abnormal category.

[0042] For the environmental factor nodes and abnormal category nodes that have no direct connection but have an indirect path, analyze the intermediate nodes and edge weights on the path to determine the association relationship.

[0043] Through the analysis of all association paths and weights in the weighted directed graph, combined with the welding abnormalities caused by the mutual influence of environmental factors, normalize the weights of direct and indirect associations, convert the weights of all environmental factors related to a certain abnormal category to the same scale, assign different weight coefficients according to the importance of each factor, and calculate the comprehensive score.

[0044] According to the historical data, set the threshold of the comprehensive influence of environmental factors for each abnormal category, compare the comprehensive scores, and exclude the abnormalities caused by environmental factors in the abnormal multi-modal welding data.

[0045] In a second aspect, the present invention provides an intelligent image data recognition and analysis system based on an AI model, including:

[0046] Data acquisition and preprocessing module: including a data acquisition unit and a real-time multimodal welding data construction unit; wherein, the data acquisition unit acquires image data, sound data, physical parameter data, and environmental data during the welding process of the welding robot, and the real-time multimodal welding data construction unit performs time synchronization and data cleaning, and integrates them to form real-time multimodal welding data;

[0047] Abnormal classification model training and application module: including a deep neural network training unit and a real-time data abnormal classification unit; wherein, the deep neural network training unit collects historical sound data and historical physical parameter data, trains using a deep neural network, and performs abnormal classification; the real-time data abnormal classification unit is based on the real-time multimodal welding data, inputs sound data and physical parameter data into the deep neural network, and obtains a welding data abnormal classification set;

[0048] Abnormal recognition module: including an image data feature extraction unit, a normal feature range setting unit, and an image data abnormal recognition unit; wherein, the image data feature extraction unit extracts image data features, including overall welding area features, weld seam close-up features, and molten pool features; the normal feature range setting unit sets the normal feature range, and the image data abnormal recognition unit compares the image data features with the normal feature range to obtain abnormal multimodal welding data;

[0049] Abnormal exclusion module: including a weighted directed graph construction unit and an environmental factor abnormal exclusion unit; wherein, the weighted directed graph construction unit determines the environment-welding association relationship based on the welding data abnormal classification set and combines the corresponding environmental data, specifically the association relationship between different factors in the environmental data and different abnormal categories; the environmental factor abnormal exclusion unit excludes the abnormalities caused by environmental factors in the abnormal multimodal welding data based on the environment-welding association relationship.

[0050] In a third aspect, the present invention provides an intelligent image data recognition and analysis device based on an AI model.

[0051] Compared with the prior art, the beneficial effects of the present invention are:

[0052] 1. The present invention uses a deep neural network to train historical sound data and physical parameter data, thereby classifying abnormalities in the sound and physical parameter data in real-time multimodal welding data, obtaining a welding data abnormal classification set, and greatly improving the accuracy and comprehensiveness of monitoring.

[0053] 2. The present invention extracts the overall features of the welding area, the close-up features of the weld seam, and the molten pool features from the image data, and compares them with the set normal feature range to obtain abnormal multi-modal welding data, providing a more intuitive and richer basis for the judgment of welding abnormalities.

[0054] 3. Based on the abnormal classification set of welding data and the corresponding environmental data, the present invention uses a weighted directed graph to determine the environment-welding correlation relationship, can accurately analyze the correlation relationship between different factors in the environmental data and different abnormal categories, and accordingly exclude the abnormalities caused by environmental factors in the abnormal multi-modal welding data. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is a schematic diagram of the steps of the intelligent image data recognition and analysis method based on the AI model of the present invention;

[0056] Figure 2 is a system structure diagram of the intelligent image data recognition and analysis system based on the AI model of the present invention;

[0057] Figure 3 is a device line drawing of the intelligent image data recognition and analysis device based on the AI model of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0058] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0059] Embodiment: As Figure 1 - Figure 2 shown, the present invention provides a technical solution.

[0060] As Figure 1 shown in the schematic diagram of the steps of the intelligent image data recognition and analysis method based on the AI model of the present invention, the present invention provides an intelligent image data recognition and analysis method based on the AI model, including the following steps:

[0061] Step S100: Obtain the image data, sound data, physical parameter data, and environmental data during the welding process of the welding robot, perform time synchronization and data cleaning, and integrate them to form real-time multi-modal welding data;

[0062] Specifically, use an industrial camera to take pictures of the overall welding area, the close-up of the weld seam, and the molten pool to obtain image data; use a microphone to collect the sound signals generated during the welding process to obtain sound data;

[0063] Measure the welding current and voltage using a current sensor and a voltage sensor respectively, monitor the welding speed and wire feeding speed using a speed sensor, obtain the angle information of each joint of the welding robot using a joint angle sensor, determine the position coordinates of the robot using a position sensor, and measure the temperature of each component of the welding equipment using a temperature sensor; integrate to obtain physical parameter data;

[0064] Obtain the ambient temperature and humidity of the welding area using a temperature sensor and a humidity sensor, and obtain the light intensity of the welding area using a light sensor; integrate to obtain environmental data;

[0065] Perform time synchronization and data cleaning, and integrate to form real-time multimodal welding data.

[0066] Step S200: Collect historical sound data and historical physical parameter data, train using a deep neural network, and perform anomaly classification; based on the real-time multimodal welding data, input the sound data and physical parameter data into the deep neural network to obtain a welding data anomaly classification set;

[0067] Specifically, label the historical sound data and historical physical parameter data, and divide the data into normal samples and abnormal samples; for abnormal samples, further divide them into different types of anomalies, including welding equipment-related anomalies, welding process-related anomalies, and welding quality-related anomalies;

[0068] Use a multimodal fusion deep neural network, including an input layer, a feature extraction layer, a fusion layer, and a classification layer; in the input layer, set a sound data input layer and a physical parameter data input layer respectively. The sound data input layer receives the normalized sound feature vector, and the physical parameter data input layer receives the normalized physical parameter data; in the feature extraction layer, for sound data, use a convolutional neural network for feature extraction, and for physical parameter data, use a fully connected layer for preliminary feature transformation; in the fusion layer, splice the sound feature vector extracted by the convolutional neural network and the physical parameter feature vector output by the fully connected layer to obtain a fusion feature vector; in the classification layer, use a fully connected layer and a Softmax activation function for anomaly classification;

[0069] Perform data partitioning, train the deep neural network, perform hyperparameter tuning, and perform model evaluation and optimization.

[0070] Further, fuse the historical voice data and historical physical parameter data, and divide them into a training set, a validation set, and a test set; define a cross-entropy loss function, use Adam as the optimization algorithm to update the parameters of the deep neural network; during the training process, input the training data into the deep neural network in batches, calculate the value of the loss function, and update the parameters of the deep neural network; use grid search to evaluate the performance under different hyperparameter combinations on the validation set, and select the optimal hyperparameter combination;

[0071] Use the test set to evaluate the trained deep neural network, calculate the classification accuracy, recall rate, and F1 value, and optimize the deep neural network.

[0072] Further, based on the real-time multi-modal welding data, input the voice data and physical parameter data into the deep neural network. According to the probability output by the Softmax function, set a threshold, and the category with a probability value greater than the threshold is determined as the category to which the sample belongs; summarize the abnormal classification results to generate an abnormal classification set of welding data, including the abnormal conditions of welding data at different time points.

[0073] In a specific embodiment, 1000 welding samples are collected, and the 1000 normalized samples are divided into a training set, a validation set, and a test set according to a ratio of 7:2:1.

[0074] Construct a deep neural network for multi-modal fusion, and the network structure is as follows:

[0075] Input layer: Voice data input layer: Receive a 128-dimensional normalized voice feature vector. Physical parameter data input layer: Receive 4-dimensional normalized physical parameter data.

[0076] Feature extraction layer: Voice data: Use a convolutional neural network (CNN) containing 3 convolutional layers for feature extraction. After each convolutional layer, a ReLU activation function and a max pooling layer are connected. Finally, the output of the convolutional layer is flattened into a feature vector. Physical parameter data: Use a network containing 2 fully connected layers for preliminary feature transformation, and a ReLU activation function is connected after each fully connected layer.

[0077] Fusion layer: Concatenate the voice feature vector extracted by the convolutional neural network and the physical parameter feature vector output by the fully connected layer to obtain a fused feature vector.

[0078] Classification layer: Use a fully connected layer and a Softmax activation function for abnormal classification, and output the probabilities of 4 categories (normal, welding equipment-related abnormality, welding process-related abnormality, welding quality-related abnormality).

[0079] Define the cross-entropy loss function. Use the Adam optimization algorithm with an initial learning rate set to 0.001. Input the training data in batches of 32 into the deep neural network and train for 50 epochs. In each epoch, calculate the value of the loss function and update the parameters of the deep neural network using the Adam optimization algorithm.

[0080] Use grid search to evaluate the performance under different hyperparameter combinations on the validation set. The hyperparameters considered include: learning rate: [0.0001, 0.001, 0.01], batch size: [16, 32, 64]; After grid search, the optimal hyperparameter combination is selected as: learning rate = 0.001, batch size = 32.

[0081] Use the test set to evaluate the trained deep neural network, and calculate the classification accuracy, recall rate, and F1 value. The results are as follows:

[0082] Classification accuracy: 0.92;

[0083] Recall rate: Normal: 0.95, Welding equipment-related anomalies: 0.88, Welding process-related anomalies: 0.90, Welding quality-related anomalies: 0.91;

[0084] F1 value: Normal: 0.96, Welding equipment-related anomalies: 0.89, Welding process-related anomalies: 0.91, Welding quality-related anomalies: 0.92.

[0085] Based on real-time multimodal welding data, input the sound data and physical parameter data into the trained deep neural network. According to the probability output by the Softmax function, set the threshold to 0.5. The category with a probability value greater than 0.5 is determined as the category to which the sample belongs. Summarize the abnormal classification results to generate an abnormal classification set of welding data, which includes the abnormal conditions of welding data at different time points. For example, during the welding process on a certain day, a total of 100 real-time samples were collected. After classification, the following abnormal classification results were obtained: Normal: 85 samples, Welding equipment-related anomalies: 3 samples, Welding process-related anomalies: 6 samples, Welding quality-related anomalies: 6 samples.

[0086] Step S300: Extract the image data features, including the overall features of the welding area, the close-up features of the weld seam, and the features of the molten pool; Set the normal feature range, compare the image data features with the normal feature range, and obtain the abnormal multimodal welding data;

[0087] Specifically, the method for extracting the overall features of the welding area is as follows: Use the Canny algorithm to identify the workpiece contour, and determine the position of the workpiece in the welding area through contour matching; Use the SIFT algorithm to extract the feature points on the workpiece, and combine the three-dimensional model and camera calibration parameters to calculate the pose information of the workpiece, including the rotation angle and translation vector;

[0088] The method for extracting the close-up features of the weld is as follows: Segment and extract the weld area from the background through the region growing algorithm, and calculate the width of the weld at different positions; Use the principle of structured light measurement, combined with the image data, to obtain the height information of the weld; During the welding process, project light with a known structure onto the weld, and by taking pictures of the deformed image of the light on the weld surface, calculate the weld height according to the principle of triangulation; Calculate the root mean square roughness of the weld surface, and through the analysis of the weld surface texture, use the gray-level co-occurrence matrix method to extract the texture features;

[0089] The method for extracting the features of the molten pool is as follows: Use the morphological processing algorithm to process the molten pool image and outline the molten pool contour; Describe the shape features of the molten pool by calculating the perimeter, area and aspect ratio of the molten pool contour.

[0090] Furthermore, in the normal working state of the welding robot, extract the mean and standard deviation of the data in the overall features of the welding area, the close-up features of the weld and the features of the molten pool, and set the normal feature range; Compare the image data features with the normal feature range, mark the abnormal data, and obtain the abnormal multi-modal welding data.

[0091] In a specific embodiment, use the anny algorithm to identify the workpiece contour for each frame of image, and then determine the position of the workpiece in the welding area through contour matching. Take the upper left corner of the image as the coordinate origin, establish a two-dimensional coordinate system, and record the coordinates of the workpiece centroid as the position information.

[0092] Use the SIFT algorithm to extract the feature points on the workpiece, and combine the known three-dimensional model and camera calibration parameters to calculate the rotation angle of the workpiece (rotation around the z-axis) and translation vector . Among them, the camera internal parameter matrix , and the external parameter matrix is obtained through calibration.

[0093] Segment and extract the weld area from the background through the region growing algorithm. On the segmented weld image, calculate the weld width every 10 pixel points, and calculate the width values at 100 positions in total .

[0094] Use the principle of structured light measurement to project light with a known structure onto the weld (the projected light is fringe light, and the fringe spacing is ), capture the deformed image of the welding light on the weld surface. According to the triangulation principle, combined with the camera calibration parameters, calculate the weld height h.

[0095] Calculate the root mean square roughness of the weld surface . First, convert the weld image to a grayscale image, then select a 10×10 pixel window within the weld area, and calculate the root mean square roughness within each window. At the same time, use the gray-level co-occurrence matrix method to extract texture features and calculate eigenvalue such as contrast, correlation, energy, and entropy.

[0096] Use morphological processing algorithms (such as erosion and dilation operations) to process the molten pool image and outline the molten pool contour. Calculate the perimeter C, area A, and aspect ratio AR of the molten pool contour.

[0097] Conduct statistical analysis on the data collected during 100 normal welding processes, calculate the mean and standard deviation of each feature, and set the normal feature ranges as follows:

[0098] For the overall features of the welding area:

[0099] Regarding the workpiece position: the mean of the x coordinate is , the standard deviation is , and the normal range is ; the mean of the y coordinate is , the standard deviation is , and the normal range is .

[0100] Regarding the workpiece attitude: the mean of the rotation angle is degrees, the standard deviation is degrees, and the normal range is degrees; the mean of the translation vector is , the standard deviation is , and the normal range is ; the mean is , the standard deviation is , and the normal range is ; the mean is , the standard deviation is , and the normal range is .

[0101] For the close-up features of the weld:

[0102] Regarding the weld width: the range of the mean width at each position is , the standard deviation is , and the normal range is .

[0103] For the weld height: the average height is , and the standard deviation is , and the normal range is .

[0104] For the texture features: the root mean square roughness has an average value of , and the standard deviation is , and the normal range is ; the average contrast is , and the standard deviation is , and the normal range is ; the average correlation is , and the standard deviation is , and the normal range is ; the average energy is , and the standard deviation is , and the normal range is ; the average entropy is , and the standard deviation is , and the normal range is .

[0105] For the molten pool features:

[0106] For the molten pool perimeter: the average perimeter is pixels, and the standard deviation is pixels, and the normal range is pixels.

[0107] For the molten pool area: the average area is pixels², and the standard deviation is pixels², and the normal range is pixels².

[0108] For the aspect ratio of the molten pool: the average aspect ratio is , and the standard deviation is , and the normal range is .

[0109] During the subsequent welding process, 20 sets of image data of the welding process were collected for anomaly detection. Each collection was also 10 seconds, with a total of 600 frames of images. The above features were extracted from each frame of image and compared with the set normal feature range.

[0110] For the overall features of the welding area: During the 5th welding process, it was found that the x coordinate of the workpiece was 950 in a certain frame, exceeding the normal range, and this frame of image was marked as abnormal data in terms of the x coordinate of the workpiece position; During the 12th welding process, the rotation angle of the workpiece reached 5 degrees, exceeding the normal range, and this frame of image was marked as abnormal data in terms of the rotation angle of the workpiece posture.

[0111] For the weld seam close-up features: During the 8th welding process, the weld seam width at a certain position was 4.2 mm, exceeding the normal range. Mark the weld seam width at this position in this frame of image as abnormal data. During the 15th welding process, the weld seam height was 2.2 mm, below the normal range. Mark the weld seam height in this frame of image as abnormal data. During the 18th welding process, the root mean square roughness was 0.16, exceeding the normal range. Mark the root mean square roughness in this frame of image as abnormal data.

[0112] For the molten pool features: During the 3rd welding process, the perimeter of the molten pool was 75 pixels, below the normal range. Mark the perimeter of the molten pool in this frame of image as abnormal data. During the 10th welding process, the area of the molten pool was 1000 pixel², exceeding the normal range. Mark the area of the molten pool in this frame of image as abnormal data. During the 16th welding process, the aspect ratio of the molten pool was 1.5, exceeding the normal range. Mark the aspect ratio of the molten pool in this frame of image as abnormal data.

[0113] Step S400: Based on the welding data anomaly classification set, combined with the corresponding environmental data, use a weighted directed graph to determine the environment-welding association relationship, specifically the association relationship between different factors in the environmental data and different anomaly categories. Based on the environment-welding association relationship, exclude the anomalies caused by environmental factors in the abnormal multimodal welding data.

[0114] Specifically, create nodes for environmental temperature, humidity, and light intensity respectively to determine environmental factor nodes. Create nodes for each anomaly category in the welding data anomaly classification set to determine anomaly category nodes. For each welding anomaly category, count the corresponding environmental data conditions before or when the anomaly occurs. Calculate the weights of the edges using conditional probability;

[0115] When there is an association between a certain environmental factor and a certain anomaly category, draw a directed edge from this environmental factor node to the corresponding anomaly category node. Among them, the method for determining whether there is an association is: if the calculated weight is not 1, it indicates that there is a non-random association relationship between the two. Determine the environment-welding association relationship.

[0116] Furthermore, for the direct connection edges from the environmental factor nodes to the anomaly category nodes in the weighted directed graph, when the weight of a certain edge is greater than 1, it indicates that this environmental factor has a direct promoting effect on this anomaly category. When the weight of a certain edge is less than 1, it indicates that this environmental factor has a negative direct association with this anomaly category;

[0117] For environmental factor nodes and anomaly category nodes that have no direct connection but have an indirect path, analyze the intermediate nodes and edge weights on the path to determine the association relationship;

[0118] By analyzing all associated paths and weights in the weighted directed graph, and combining the welding abnormalities caused by the mutual influence of environmental factors, the weights of direct and indirect associations are normalized, and the weights of environmental factors related to a certain abnormality category are transformed to the same scale. Different weight coefficients are assigned according to the importance of each factor, and the comprehensive score is calculated;

[0119] According to historical data, a threshold for the comprehensive influence of environmental factors is set for each abnormality category. By comparing the comprehensive scores, the abnormalities caused by environmental factors in the abnormal multimodal welding data are excluded.

[0120] In a specific embodiment, a certain welding production line was monitored for a period of time, and an abnormal classification set of welding data and corresponding environmental data were collected. The abnormal classification set of welding data contains three abnormality categories: porosity abnormality, crack abnormality, and lack of fusion abnormality. The environmental data includes environmental temperature, humidity, and light intensity. A total of 1000 welding sample data were collected, including 700 normal samples and 300 abnormal samples. The specific distribution of abnormal samples is as follows: 100 porosity abnormalities, 100 crack abnormalities, and 100 lack of fusion abnormalities. According to the calculated edge weights, since all edge weights are not 1, directed edges are drawn from each environmental factor node to the corresponding abnormality category node to form a weighted directed graph.

[0121] For the porosity abnormality, the edge weight from the "temperature node" to the "porosity abnormality node" , indicating that an environmental temperature higher than 30°C has a direct promoting effect on the porosity abnormality; the edge weight from the "humidity node" to the "porosity abnormality node" , indicating that a humidity higher than 60% has a direct promoting effect on the porosity abnormality; the edge weight from the "light intensity node" to the "porosity abnormality node" , indicating that a light intensity lower than 50 lux has a direct promoting effect on the porosity abnormality.

[0122] For the crack abnormality, the edge weight from the "temperature node" to the "crack abnormality node" , indicating that an environmental temperature lower than 10°C has a direct promoting effect on the crack abnormality; the edge weight from the "humidity node" to the "crack abnormality node" , indicating that a humidity lower than 30% has a direct promoting effect on the crack abnormality; the edge weight from the "light intensity node" to the "crack abnormality node" , indicating that a light intensity higher than 200 lux has a negative direct association with the crack abnormality.

[0123] For the lack of fusion abnormality, the edge weight from the "temperature node" to the "lack of fusion abnormality node" , indicating that an environmental temperature between 20 - 25°C has a direct promoting effect on the lack of fusion abnormality; the edge weight from the "humidity node" to the "lack of fusion abnormality node" , indicating that humidity between 40 - 50% has a direct promoting effect on lack of fusion abnormalities; the edge weight from the "light intensity node" to the "lack of fusion abnormality node" , indicating that light intensity between 80 - 120 lux has a negative direct correlation with lack of fusion abnormalities.

[0124] There is an indirect path: "temperature node" - "coefficient of thermal expansion change of metal node" - "crack abnormality node". The edge weight from the "temperature node" to the "coefficient of thermal expansion change of metal node" is 1.5, and the edge weight from the "coefficient of thermal expansion change of metal node" to the "crack abnormality node" is 1.2. Then the combined weight of this indirect path is , indicating that temperature indirectly promotes crack abnormalities by affecting the coefficient of thermal expansion change of metal.

[0125] Normalize the weights of direct and indirect correlations using the maximum - minimum normalization method. Taking pore abnormalities as an example, the weights of direct correlations are respectively , , . The normalized weights are respectively , , .

[0126] According to experience, the weight coefficients assigned to temperature, humidity, and light intensity are 0.5, 0.3, and 0.2 respectively. Then the comprehensive score of pore abnormalities .

[0127] Similarly, calculate the comprehensive scores of crack abnormalities and lack of fusion abnormalities. The comprehensive score of crack abnormalities , and the comprehensive score of lack of fusion abnormalities .

[0128] According to historical data, set the threshold for the comprehensive influence of environmental factors for pore abnormalities to be 0.8, for crack abnormalities to be 0.75, and for lack of fusion abnormalities to be 0.65. Since , , , it can be judged that the currently detected pore abnormalities, crack abnormalities, and lack of fusion abnormalities are all caused by environmental factors, and these abnormalities are excluded from the abnormal multi - modal welding data.

[0129] Such as Figure 2 Based on the system structure diagram of the intelligent image data recognition and analysis system of the AI model, the present invention provides an intelligent image data recognition and analysis system based on the AI model, including:

[0130] Data acquisition and preprocessing module: It includes a data acquisition unit and a real-time multimodal welding data construction unit. Among them, the data acquisition unit acquires image data, sound data, physical parameter data, and environmental data during the welding process of the welding robot. The real-time multimodal welding data construction unit performs time synchronization and data cleaning, and integrates them into real-time multimodal welding data.

[0131] Abnormal classification model training and application module: It includes a deep neural network training unit and a real-time data abnormal classification unit. Among them, the deep neural network training unit collects historical sound data and historical physical parameter data, trains using a deep neural network, and performs abnormal classification. The real-time data abnormal classification unit, based on the real-time multimodal welding data, inputs sound data and physical parameter data into the deep neural network to obtain a welding data abnormal classification set.

[0132] Abnormal recognition module: It includes an image data feature extraction unit, a normal feature range setting unit, and an image data abnormal recognition unit. Among them, the image data feature extraction unit extracts image data features, including the overall feature of the welding area, the close-up feature of the weld seam, and the feature of the molten pool. The normal feature range setting unit sets the normal feature range. The image data abnormal recognition unit compares the image data features with the normal feature range to obtain abnormal multimodal welding data.

[0133] Abnormal elimination module: It includes a weighted directed graph construction unit and an environmental factor abnormal elimination unit. Among them, the weighted directed graph construction unit, based on the welding data abnormal classification set, combines the corresponding environmental data, and uses a weighted directed graph to determine the environment-welding association relationship, specifically the association relationship between different factors in the environmental data and different abnormal categories. The environmental factor abnormal elimination unit eliminates the abnormalities caused by environmental factors in the abnormal multimodal welding data based on the environment-welding association relationship.

[0134] As Figure 3 As shown in the device line drawing of the intelligent image data recognition and analysis device based on the AI model of the present invention, the present invention provides an intelligent image data recognition and analysis device based on the AI model.

[0135] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present invention. Any reference signs in the claims should not be regarded as limiting the claims involved.

Claims

1. An intelligent image data recognition and analysis method based on an AI model, characterized in that It includes the following steps: Obtain the image data, sound data, physical parameter data, and environmental data during the welding process of the welding robot, perform time synchronization and data cleaning, and integrate them to form real-time multimodal welding data; Collect historical sound data and historical physical parameter data, use a deep neural network for training, and perform anomaly classification; Based on the real-time multimodal welding data, input the sound data and physical parameter data into the deep neural network to obtain an abnormal classification set of welding data; Extract the image data features, including the overall features of the welding area, the close-up features of the weld seam, and the molten pool features; Set the normal feature range, compare the image data features with the normal feature range, and obtain abnormal multimodal welding data; Based on the abnormal classification set of welding data, combined with the corresponding environmental data, use a weighted directed graph to determine the environment-welding correlation relationship, specifically the correlation relationship between different factors in the environmental data and different abnormal categories; Based on the environment-welding correlation relationship, exclude the anomalies caused by environmental factors in the abnormal multimodal welding data.

2. The intelligent image data recognition and analysis method based on the AI model according to claim 1, wherein The obtaining of the image data, sound data, physical parameter data, and environmental data during the welding process of the welding robot, performing time synchronization and data cleaning, and integrating them to form real-time multimodal welding data includes: Use an industrial camera to take pictures of the overall welding area, the close-up of the weld seam, and the molten pool to obtain image data; Use a microphone to collect the sound signals generated during the welding process to obtain sound data; Use a current sensor and a voltage sensor to measure the welding current and voltage respectively, use a speed sensor to monitor the welding speed and wire feeding speed, use a joint angle sensor to obtain the angle information of each joint of the welding robot, use a position sensor to determine the position coordinates of the robot, and use a temperature sensor to measure the temperature of each component of the welding equipment; Integrate to obtain physical parameter data; Use a temperature sensor and a humidity sensor to obtain the environmental temperature and humidity of the welding area, and use a light sensor to obtain the light intensity of the welding area; Integrate to obtain environmental data; Perform time synchronization and data cleaning, and integrate to form real-time multimodal welding data.

3. The intelligent image data recognition and analysis method based on the AI model according to claim 1, wherein The collecting of historical sound data and historical physical parameter data, using a deep neural network for training, and performing anomaly classification includes: Label the historical sound data and historical physical parameter data, and divide the data into normal samples and abnormal samples; For abnormal samples, further divide them into different types of anomalies, including welding equipment-related anomalies, welding process-related anomalies, and welding quality-related anomalies; A deep neural network using multi-modal fusion, including an input layer, a feature extraction layer, a fusion layer, and a classification layer; in the input layer, a sound data input layer and a physical parameter data input layer are respectively set, the sound data input layer receives the normalized sound feature vector, and the physical parameter data input layer receives the normalized physical parameter data; in the feature extraction layer, for sound data, a convolutional neural network is used for feature extraction, and for physical parameter data, a fully connected layer is used for preliminary feature transformation; in the fusion layer, the sound feature vector extracted by the convolutional neural network and the physical parameter feature vector output by the fully connected layer are concatenated to obtain a fusion feature vector; in the classification layer, a fully connected layer and a Softmax activation function are used for anomaly classification; Perform data partitioning, train the deep neural network, perform hyperparameter tuning, and perform model evaluation and optimization; Fuse historical sound data and historical physical parameter data, and divide them into a training set, a validation set, and a test set; define a cross-entropy loss function, use Adam as the optimization algorithm to update the parameters of the deep neural network; during training, input the training data into the deep neural network in batches, calculate the value of the loss function, and update the parameters of the deep neural network; use grid search to evaluate the performance under different hyperparameter combinations on the validation set and select the optimal hyperparameter combination; Use the test set to evaluate the trained deep neural network, calculate the classification accuracy, recall rate, and F1 value, and optimize the deep neural network.

4. The intelligent image data recognition and analysis method based on the AI model according to claim 1, wherein Based on the real-time multi-modal welding data, input the sound data and physical parameter data into the deep neural network to obtain a welding data anomaly classification set, including: Based on the real-time multi-modal welding data, input the sound data and physical parameter data into the deep neural network. According to the probability output by the Softmax function, set a threshold, and the category with a probability value greater than the threshold is determined as the category to which the sample belongs; summarize the anomaly classification results to generate a welding data anomaly classification set, which contains the welding data anomaly situations at different time points.

5. The intelligent image data recognition and analysis method based on the AI model according to claim 1, wherein The extraction of image data features, including the overall features of the welding area, the close-up features of the weld seam, and the features of the molten pool, includes: The method for extracting the overall features of the welding area is: use the Canny algorithm to identify the workpiece contour, and determine the position of the workpiece in the welding area through contour matching; use the SIFT algorithm to extract the feature points on the workpiece, and combine the three-dimensional model and the camera calibration parameters to calculate the pose information of the workpiece, including the rotation angle and the translation vector; The method for extracting the close-up features of the weld seam is: segment and extract the weld seam area from the background through the region growing algorithm, and calculate the width of the weld seam at different positions; use the principle of structured light measurement, combined with the image data, to obtain the height information of the weld seam; during the welding process, project light with a known structure onto the weld seam, and take a picture of the deformed image of the light on the weld seam surface, and calculate the height of the weld seam according to the principle of triangulation; calculate the root mean square roughness of the weld seam surface, and extract the texture features by analyzing the texture of the weld seam surface using the gray-level co-occurrence matrix method; The method for extracting the molten pool features is as follows: Use the morphological processing algorithm to process the molten pool image and outline the molten pool contour; describe the shape features of the molten pool by calculating the perimeter, area, and aspect ratio of the molten pool contour.

6. The intelligent image data recognition and analysis method based on the AI model according to claim 1, wherein Set the normal feature range, compare the image data features with the normal feature range, and obtain abnormal multimodal welding data, including: Under the normal working state of the welding robot, extract the mean and standard deviation of the data in the overall features of the welding area, the close-up features of the weld seam, and the molten pool features, and set the normal feature range; compare the image data features with the normal feature range, mark the abnormal data, and obtain abnormal multimodal welding data.

7. The intelligent image data recognition and analysis method based on the AI model according to claim 1, characterized in that, Based on the abnormal classification set of welding data, combined with the corresponding environmental data, use a weighted directed graph to determine the environment-welding association relationship, specifically the association relationship between different factors in the environmental data and different abnormal categories, including: Create nodes for the environmental temperature, humidity, and light intensity respectively to determine the environmental factor nodes; create nodes for each abnormal category in the abnormal classification set of welding data to determine the abnormal category nodes; for each welding abnormal category, count the corresponding environmental data situation before or when the abnormality occurs; calculate the weight of the edge using conditional probability. When there is an association between a certain environmental factor and a certain abnormal category, draw a directed edge from the environmental factor node to the corresponding abnormal category node. Among them, the method for judging whether there is an association is: if the calculated weight is not 1, it indicates that there is a non-random association relationship between the two; determine the environment-welding association relationship.

8. The intelligent image data recognition and analysis method based on the AI model according to claim 7, wherein Based on the environment-welding association relationship, exclude the abnormalities caused by environmental factors in the abnormal multimodal welding data, including: For the direct connection edge from the environmental factor node to the abnormal category node in the weighted directed graph, when the weight of a certain edge is greater than 1, it indicates that the environmental factor has a direct promoting effect on the abnormal category, and when the weight of a certain edge is less than 1, it indicates that the environmental factor has a negative direct association with the abnormal category. For the environmental factor nodes and abnormal category nodes that have no direct connection but have an indirect path, analyze the intermediate nodes and edge weights on the path to determine the association relationship. Through the analysis of all association paths and weights in the weighted directed graph, combined with the welding abnormalities caused by the mutual influence of environmental factors, normalize the weights of direct and indirect associations, convert the weights of all environmental factors related to a certain abnormal category to the same scale, assign different weight coefficients according to the importance of each factor, and calculate the comprehensive score. According to the historical data, set the threshold of the comprehensive influence of environmental factors for each abnormal category, compare the comprehensive scores, and exclude the abnormalities caused by environmental factors in the abnormal multimodal welding data.

9. An intelligent image data recognition and analysis system based on an AI model, which uses the intelligent image data recognition and analysis method based on an AI model described in any one of claims 1-8, characterized in that, Including: Data acquisition and preprocessing module: including: data acquisition unit and real-time multimodal welding data construction unit; among them, the data acquisition unit obtains the image data, sound data, physical parameter data, and environmental data of the welding robot during the welding process, and the real-time multimodal welding data construction unit performs time synchronization and data cleaning and integrates them into real-time multimodal welding data. Abnormal classification model training and application module: including: a deep neural network training unit and a real-time data abnormal classification unit; among them, the deep neural network training unit collects historical sound data and historical physical parameter data, uses the deep neural network for training, and conducts abnormal classification; the real-time data abnormal classification unit is based on real-time multimodal welding data, inputs sound data and physical parameter data into the deep neural network, and obtains a welding data abnormal classification set; Abnormal recognition module: including: an image data feature extraction unit, a normal feature range setting unit, and an image data abnormal recognition unit; among them, the image data feature extraction unit extracts image data features, including the overall features of the welding area, the close-up features of the weld seam, and the features of the molten pool; the normal feature range setting unit sets the normal feature range, and the image data abnormal recognition unit compares the image data features with the normal feature range to obtain abnormal multimodal welding data; Abnormal elimination module: including: a weighted directed graph construction unit and an environmental factor abnormal elimination unit; among them, the weighted directed graph construction unit is based on the welding data abnormal classification set, combines the corresponding environmental data, and uses the weighted directed graph to determine the environment-welding association relationship, specifically the association relationship between different factors in the environmental data and different abnormal categories; the environmental factor abnormal elimination unit eliminates the abnormalities caused by environmental factors in the abnormal multimodal welding data based on the environment-welding association relationship.

10. An intelligent image data recognition and analysis device based on an AI model, characterized in that, The device includes the intelligent image data recognition and analysis system based on the AI model described in claim 9 and an embedded computer.

Images