Intelligent image data identification and analysis method, system and device based on AI model

Abstract

The invention discloses an intelligent image data identification and analysis method, system and device based on an AI model, and belongs to the technical field of image data analysis. Image data, sound data, physical parameter data and environment data of the welding robot in the welding process are obtained and integrated to form real-time multi-mode welding data; training historical data by using a deep neural network, and carrying out anomaly classification; inputting the sound data and the physical parameter data into a deep neural network to obtain a welding data anomaly classification set; extracting image data features, setting a normal feature range, and comparing the image data features with the normal feature range to obtain abnormal multi-mode welding data; determining an environment-welding association relationship by using a weighted directed graph, wherein the environment-welding association relationship is specifically an association relationship between different factors and different exception categories in the environment data; and based on the environment-welding incidence relation, abnormities caused by environment factors in the abnormal multi-mode welding data are eliminated.

Description

Technical Field

[0001] The present invention relates to the field of image data analysis technology, and specifically to an intelligent image data recognition and analysis method, system and device based on an AI model. Background Art

[0002] In modern industrial production, welding, as a key connection process, is widely used in many fields such as automobile manufacturing, aerospace, shipbuilding, etc. With the development of industrial automation and intelligence, the requirements for welding quality and production efficiency are increasing.

[0003] Existing abnormality judgment methods are mostly based on simple threshold settings. When the monitoring data exceeds or falls below the set threshold, it is judged as abnormal, which is prone to misjudgment and missed judgment, and cannot adapt to diverse welding conditions. The welding process is affected by a variety of factors, but the existing technology lacks in-depth analysis of the correlation between these factors and cannot accurately determine the root cause of the abnormality. Although some monitoring systems have introduced image data, the processing and analysis of image data is relatively rudimentary, and only simple image recognition can be performed. Summary of the invention

[0004] The purpose of the present invention is to provide an intelligent image data recognition and analysis method, system and device based on an AI model to solve the problems raised in the prior art.

[0005] To achieve the above object, the present invention provides the following technical solutions: In a first aspect, the present invention provides an intelligent image data recognition and analysis method based on an AI model, comprising the following steps: Acquire the image data, sound data, physical parameter data and environmental data of the welding robot during the welding process, perform time synchronization and data cleaning, and integrate them into real-time multi-modal welding data; Collect historical sound data and historical physical parameter data, use deep neural network for training, and perform abnormal classification; based on real-time multimodal welding data, input sound data and physical parameter data into the deep neural network to obtain the welding data abnormal classification set; Extract image data features, including overall features of the welding area, close-up features of the weld, and features of the molten pool; set a normal feature range, compare the image data features with the normal feature range, and obtain abnormal multimodal welding data; Based on the abnormal classification set of welding data and the corresponding environmental data, a weighted directed graph is used to determine the environment-welding association relationship, specifically the association relationship between different factors in the environmental data and different abnormal categories; based on the environment-welding association relationship, anomalies caused by environmental factors in abnormal multimodal welding data are eliminated.

[0006] In combination with the first aspect, in a first implementation of the first aspect of the present application, the image data, sound data, physical parameter data and environmental data of the welding robot during the welding process are acquired, time synchronization and data cleaning are performed, and integrated to form real-time multimodal welding data, including: Use an industrial camera to capture the entire welding area, close-ups of the weld, and the molten pool to obtain image data; use a microphone to collect sound signals generated during the welding process to obtain sound data; Use current sensors and voltage sensors to measure welding current and voltage respectively, use speed sensors to monitor welding speed and wire feeding speed, use joint angle sensors to obtain angle information of each joint of the welding robot, use position sensors to determine the position coordinates of the robot, and use temperature sensors to measure the temperature of each component of the welding equipment; integrate to obtain physical parameter data; Use temperature sensors and humidity sensors to obtain the ambient temperature and humidity of the welding area, and use light sensors to obtain the light intensity of the welding area; integrate and obtain environmental data; Perform time synchronization and data cleaning to integrate and form real-time multimodal welding data.

[0007] In combination with the first aspect, in a second implementation of the first aspect of the present application, the collecting of historical sound data and historical physical parameter data, training using a deep neural network, and performing abnormal classification include: Label the historical sound data and historical physical parameter data, and divide the data into normal samples and abnormal samples; for abnormal samples, further divide them into different types of abnormalities, including welding equipment-related abnormalities, welding process-related abnormalities, and welding quality-related abnormalities; A deep neural network using multimodal fusion includes an input layer, a feature extraction layer, a fusion layer and a classification layer; in the input layer, a sound data input layer and a physical parameter data input layer are respectively set, the sound data input layer receives a normalized sound feature vector, and the physical parameter data input layer receives normalized physical parameter data; in the feature extraction layer, a convolutional neural network is used to extract features for sound data, and a fully connected layer is used to perform preliminary feature transformation for physical parameter data; in the fusion layer, the sound feature vector extracted by the convolutional neural network and the physical parameter feature vector output by the fully connected layer are concatenated to obtain a fused feature vector; in the classification layer, a fully connected layer and a Softmax activation function are used to perform abnormal classification; Perform data partitioning, train deep neural networks, adjust hyperparameters, and evaluate and optimize models.

[0008] In combination with the first aspect, in a third implementation of the first aspect of the present application, the data segmentation, deep neural network training, hyperparameter adjustment, and model evaluation and optimization include: The historical sound data and the historical physical parameter data are integrated and divided into training set, validation set and test set; the cross entropy loss function is defined, and Adam is used as the optimization algorithm to update the parameters of the deep neural network; during the training process, the training data is input into the deep neural network in batches, the value of the loss function is calculated, and the parameters of the deep neural network are updated; grid search is used to evaluate the performance under different hyperparameter combinations on the validation set and select the optimal hyperparameter combination; Use the test set to evaluate the trained deep neural network, calculate the classification accuracy, recall rate and F1 value, and optimize the deep neural network.

[0009] In combination with the first aspect, in a fourth implementation of the first aspect of the present application, based on real-time multimodal welding data, inputting sound data and physical parameter data into a deep neural network to obtain a welding data abnormality classification set includes: Based on real-time multimodal welding data, sound data and physical parameter data are input into the deep neural network. According to the probability output by the Softmax function, a threshold is set, and the category with a probability value greater than the threshold is judged as the category to which the sample belongs. The abnormal classification results are summarized to generate a welding data abnormal classification set, which includes abnormal welding data at different time points.

[0010] In combination with the first aspect, in a fifth implementation of the first aspect of the present application, the extracted image data features, including overall features of the welding area, close-up features of the weld, and features of the molten pool, include: The method for extracting the overall features of the welding area is as follows: using the Canny algorithm to identify the workpiece contour, and determining the position of the workpiece in the welding area through contour matching; using the SIFT algorithm to extract the feature points on the workpiece, combining the three-dimensional model and camera calibration parameters, and calculating the workpiece posture information, including the rotation angle and translation vector; Assume that the point set on the workpiece contour is , find the position that is most similar to the standard template contour through the contour matching algorithm, and obtain the position coordinates of the workpiece in this coordinate system , here That is the position parameter of the workpiece in the welding area.

[0011] The SIFT algorithm is used to extract feature points on the workpiece. Suppose the extracted feature point set is Combining the 3D model and camera calibration parameters, the camera’s intrinsic parameter matrix is ,in , is the component of focal length in x and y directions, , is the image center coordinate. The external parameter matrix is , R is the rotation matrix, t is the translation vector. The posture information of the workpiece is calculated through feature point matching and triangulation principle. The rotation angle θ is calculated by the elements of the rotation matrix R, for example, the rotation angle around the z axis is ; Translation vector Directly given by the external parameter matrix, these parameters accurately describe the posture of the workpiece and provide a basis for determining whether the workpiece is displaced or its posture changes during the welding process.

[0012] The method for extracting close-up features of welds is as follows: the weld area is segmented and extracted from the background by a region growing algorithm, and the width of the weld at different positions is calculated; the height information of the weld is obtained by combining the structured light measurement principle with the image data; during the welding process, light of known structure is projected onto the weld, and the weld height is calculated according to the triangulation principle by taking the deformed image of the light on the weld surface; the root mean square roughness of the weld surface is calculated, and the texture features are extracted by using the grayscale co-occurrence matrix method through the analysis of the weld surface texture; After the weld area is segmented and extracted from the background by the region growing algorithm, the pixel point set of the weld area in the image is In the image coordinate system, the weld width at different positions is calculated. For a certain position i, by finding the two edge points perpendicular to the weld direction at that position and , then the weld width at this position , the unit is pixel, and then it is converted into the actual weld width size according to the conversion relationship between the camera calibrated pixels and the actual size.

[0013] Using the structured light measurement principle, the coordinates of the structured light pattern projected onto the weld surface on the camera imaging plane are set as According to the triangulation principle, given the relative position relationship between the structured light projector and the camera (baseline distance b) and the camera's intrinsic parameter matrix K, for a certain pixel point , the corresponding three-dimensional space point Calculated by the following formula: , combined with the geometric information of the structured light pattern and the known projection angle ,pass Calculate the height of the point , thereby obtaining the height information of the weld.

[0014] Calculate the RMS roughness of the weld surface , let the pixel gray value matrix of a local area on the weld surface be , RMS roughness ,in is the average grayscale value of the pixels in the local area. The grayscale co-occurrence matrix method is used to extract texture features. Let the grayscale co-occurrence matrix be , L is the gray level. By calculating the eigenvalues ​​of the gray level co-occurrence matrix, such as contrast, correlation, energy, entropy, etc., the texture characteristics of the weld surface are described. These texture characteristics can be used to judge the quality of the weld and whether there are defects.

[0015] The method for extracting the molten pool features is: using a morphological processing algorithm to process the molten pool image and outline the molten pool contour; and describing the shape characteristics of the molten pool by calculating the perimeter, area and aspect ratio of the molten pool contour.

[0016] Assume that the point set on the molten pool contour is , calculate the perimeter of the melt pool contour ,area ; Aspect ratio ,in and are the length and width of the circumscribed rectangle of the molten pool outline, respectively. These parameters comprehensively describe the shape characteristics of the molten pool and provide a quantitative basis for judging whether the molten pool state is normal.

[0017] In combination with the first aspect, in a sixth implementation of the first aspect of the present application, setting a normal feature range, comparing the image data features with the normal feature range, and obtaining abnormal multimodal welding data includes: When the welding robot is in normal working condition, the mean and standard deviation of the data in the overall features of the welding area, close-up features of the weld and molten pool features are extracted, and the normal feature range is set; the image data features are compared with the normal feature range, the abnormal data are marked, and the abnormal multimodal welding data is obtained.

[0018] In combination with the first aspect, in a seventh implementation of the first aspect of the present application, the environment-welding association relationship is determined using a weighted directed graph based on the abnormal classification set of welding data and in combination with the corresponding environmental data, specifically the association relationship between different factors in the environmental data and different abnormal categories, including: Create nodes for ambient temperature, humidity and light intensity respectively, and determine the environmental factor nodes; create nodes for each abnormal category in the welding data abnormal classification set, and determine the abnormal category nodes; for each welding abnormal category, count the corresponding environmental data before or when the abnormality occurs; use conditional probability to calculate the weight of the edge; When an environmental factor is associated with an abnormal category, a directed edge is drawn from the environmental factor node to the corresponding abnormal category node, wherein the method for determining whether there is an association is as follows: if the calculated weight is not 1, it indicates that there is a non-random association between the two; and the environment-welding association is determined.

[0019] In combination with the first aspect, in an eighth implementation of the first aspect of the present application, the method of eliminating anomalies caused by environmental factors in abnormal multimodal welding data based on the environment-welding association relationship includes: For the direct connection edge from the environmental factor node to the abnormal category node in the weighted directed graph, when the weight of an edge is greater than 1, it indicates that the environmental factor has a direct promoting effect on the abnormal category; when the weight of an edge is less than 1, it indicates that the environmental factor has a negative direct correlation with the abnormal category; For environmental factor nodes and abnormal category nodes that have no direct connection but indirect paths, the intermediate nodes and edge weights on the path are analyzed to determine the association relationship; By analyzing all the associated paths and weights in the weighted directed graph, combined with the welding anomalies caused by the mutual influence of environmental factors, the weights of direct and indirect associations are normalized, and the weights of all environmental factors related to a certain anomaly category are converted to the same scale. Different weight coefficients are assigned according to the importance of each factor, and the comprehensive score is calculated; According to historical data, a threshold for the comprehensive impact of environmental factors is set for each abnormal category, and the comprehensive scores are compared to eliminate the anomalies caused by environmental factors in the abnormal multimodal welding data.

[0020] In a second aspect, the present invention provides an intelligent image data recognition and analysis system based on an AI model, comprising: Data acquisition and preprocessing module: including: data acquisition unit and real-time multi-modal welding data construction unit; wherein, the data acquisition unit obtains the image data, sound data, physical parameter data and environmental data of the welding robot during the welding process, and the real-time multi-modal welding data construction unit performs time synchronization and data cleaning, and integrates to form real-time multi-modal welding data; Abnormal classification model training and application module: including: deep neural network training unit and real-time data abnormal classification unit; wherein, the deep neural network training unit collects historical sound data and historical physical parameter data, uses deep neural network for training, and performs abnormal classification; the real-time data abnormal classification unit inputs sound data and physical parameter data into the deep neural network based on real-time multimodal welding data, and obtains the welding data abnormal classification set; Abnormal identification module: including: image data feature extraction unit, normal feature range setting unit and image data abnormal identification unit; wherein the image data feature extraction unit extracts image data features, including overall features of the welding area, close-up features of the weld and molten pool features; the normal feature range setting unit sets the normal feature range, and the image data abnormal identification unit compares the image data features with the normal feature range to obtain abnormal multi-modal welding data; Abnormality elimination module: including: a weighted directed graph construction unit and an environmental factor abnormality elimination unit; wherein, the weighted directed graph construction unit is based on the welding data abnormality classification set, combined with the corresponding environmental data, and uses a weighted directed graph to determine the environment-welding association relationship, specifically the association relationship between different factors in the environmental data and different abnormal categories; the environmental factor abnormality elimination unit eliminates abnormalities caused by environmental factors in abnormal multimodal welding data based on the environment-welding association relationship.

[0021] In a third aspect, the present invention provides an intelligent image data recognition and analysis device based on an AI model.

[0022] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention uses a deep neural network to train historical sound data and physical parameter data, thereby performing abnormal classification on the sound and physical parameter data in real-time multimodal welding data, and obtaining an abnormal classification set of welding data, which greatly improves the accuracy and comprehensiveness of monitoring.

[0023] 2. The present invention obtains abnormal multimodal welding data by extracting the overall characteristics of the welding area, close-up characteristics of the weld and molten pool characteristics of the image data, and comparing them with the set normal feature range, thereby providing a more intuitive and richer basis for judging welding abnormalities.

[0024] 3. The present invention is based on the welding data anomaly classification set and the corresponding environmental data, and uses a weighted directed graph to determine the environment-welding association relationship. It can accurately analyze the association between different factors in the environmental data and different anomaly categories, and accordingly eliminate the anomalies caused by environmental factors in the abnormal multimodal welding data. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of the steps of the intelligent image data recognition and analysis method based on the AI ​​model of the present invention; Figure 2 It is a system structure diagram of the intelligent image data recognition and analysis system based on the AI ​​model of the present invention; Figure 3 It is a line drawing of the intelligent image data recognition and analysis device based on the AI ​​model of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] Example: Figure 1-Figure 2 As shown, the present invention provides a technical solution. like Figure 1 As shown in the schematic diagram of the steps of the intelligent image data recognition and analysis method based on the AI ​​model of the present invention, the present invention provides an intelligent image data recognition and analysis method based on the AI ​​model, comprising the following steps: Step S100: acquiring image data, sound data, physical parameter data and environmental data of the welding robot during the welding process, performing time synchronization and data cleaning, and integrating to form real-time multi-modal welding data; Specifically, an industrial camera is used to capture the entire welding area, close-up of the weld, and the molten pool to obtain image data; a microphone is used to collect sound signals generated during the welding process to obtain sound data; Use current sensors and voltage sensors to measure welding current and voltage respectively, use speed sensors to monitor welding speed and wire feeding speed, use joint angle sensors to obtain angle information of each joint of the welding robot, use position sensors to determine the position coordinates of the robot, and use temperature sensors to measure the temperature of each component of the welding equipment; integrate to obtain physical parameter data; Use temperature sensors and humidity sensors to obtain the ambient temperature and humidity of the welding area, and use light sensors to obtain the light intensity of the welding area; integrate and obtain environmental data; Perform time synchronization and data cleaning to integrate and form real-time multimodal welding data.

[0028] Step S200: collecting historical sound data and historical physical parameter data, using a deep neural network for training, and performing abnormal classification; based on real-time multimodal welding data, inputting sound data and physical parameter data into the deep neural network to obtain a welding data abnormal classification set; Specifically, the historical sound data and historical physical parameter data are annotated and the data are divided into normal samples and abnormal samples; the abnormal samples are further divided into different types of abnormalities, including welding equipment-related abnormalities, welding process-related abnormalities, and welding quality-related abnormalities; A deep neural network using multimodal fusion includes an input layer, a feature extraction layer, a fusion layer and a classification layer; in the input layer, a sound data input layer and a physical parameter data input layer are respectively set, the sound data input layer receives a normalized sound feature vector, and the physical parameter data input layer receives normalized physical parameter data; in the feature extraction layer, a convolutional neural network is used to extract features for sound data, and a fully connected layer is used to perform preliminary feature transformation for physical parameter data; in the fusion layer, the sound feature vector extracted by the convolutional neural network and the physical parameter feature vector output by the fully connected layer are concatenated to obtain a fused feature vector; in the classification layer, a fully connected layer and a Softmax activation function are used to perform abnormal classification; Perform data partitioning, train deep neural networks, adjust hyperparameters, and evaluate and optimize models.

[0029] Furthermore, the historical sound data and historical physical parameter data are fused and divided into training set, validation set and test set; the cross entropy loss function is defined, and Adam is used as the optimization algorithm to update the parameters of the deep neural network; during the training process, the training data is input into the deep neural network in batches, the value of the loss function is calculated, and the parameters of the deep neural network are updated; grid search is used to evaluate the performance under different hyperparameter combinations on the validation set, and the optimal hyperparameter combination is selected; Use the test set to evaluate the trained deep neural network, calculate the classification accuracy, recall rate and F1 value, and optimize the deep neural network.

[0030] Furthermore, based on real-time multimodal welding data, sound data and physical parameter data are input into the deep neural network. According to the probability output by the Softmax function, a threshold is set, and the category with a probability value greater than the threshold is judged as the category to which the sample belongs. The abnormal classification results are summarized to generate a welding data abnormal classification set, which includes abnormal welding data at different time points.

[0031] In a specific embodiment, data is collected from 1000 welding samples, and the 1000 normalized samples are divided into a training set, a validation set, and a test set in a ratio of 7:2:1.

[0032] Construct a deep neural network for multimodal fusion. The network structure is as follows: Input layer: Sound data input layer: receives 128-dimensional normalized sound feature vectors. Physical parameter data input layer: receives 4-dimensional normalized physical parameter data.

[0033] Feature extraction layer: Sound data: A convolutional neural network (CNN) with 3 convolutional layers is used for feature extraction. Each convolutional layer is followed by a ReLU activation function and a max pooling layer. Finally, the output of the convolutional layer is flattened into a feature vector. Physical parameter data: A network with 2 fully connected layers is used for preliminary feature transformation, each of which is followed by a ReLU activation function.

[0034] Fusion layer: concatenates the sound feature vector extracted by the convolutional neural network and the physical parameter feature vector output by the fully connected layer to obtain a fused feature vector.

[0035] Classification layer: Use a fully connected layer and Softmax activation function to perform anomaly classification and output the probabilities of four categories (normal, welding equipment-related anomalies, welding process-related anomalies, and welding quality-related anomalies).

[0036] Define the cross entropy loss function. Use the Adam optimization algorithm and set the initial learning rate to 0.001. Input the training data into the deep neural network in batches of 32 and train for 50 epochs. In each epoch, calculate the value of the loss function and use the Adam optimization algorithm to update the parameters of the deep neural network.

[0037] Grid search is used to evaluate the performance of different hyperparameter combinations on the validation set. The hyperparameters considered include: learning rate: [0.0001, 0.001, 0.01], batch size: [16, 32, 64]; after grid search, the optimal hyperparameter combination is selected as: learning rate = 0.001, batch size = 32.

[0038] The trained deep neural network is evaluated using the test set to calculate the classification accuracy, recall rate, and F1 value. The results are as follows: Classification accuracy: 0.92; Recall rate: normal: 0.95, welding equipment related abnormality: 0.88, welding process related abnormality: 0.90, welding quality related abnormality: 0.91; F1 value: Normal: 0.96, abnormalities related to welding equipment: 0.89, abnormalities related to welding process: 0.91, abnormalities related to welding quality: 0.92.

[0039] Based on real-time multimodal welding data, sound data and physical parameter data are input into the trained deep neural network, and the threshold is set to 0.5 according to the probability output by the Softmax function. The category with a probability value greater than 0.5 is determined to be the category to which the sample belongs. The abnormal classification results are summarized to generate a welding data abnormal classification set, which contains abnormal welding data at different time points. For example, during the welding process of a certain day, a total of 100 real-time samples were collected. After classification, the following abnormal classification results were obtained: normal: 85 samples, welding equipment-related abnormalities: 3 samples, welding process-related abnormalities: 6 samples, welding quality-related abnormalities: 6 samples.

[0040] Step S300: extracting image data features, including overall features of the welding area, close-up features of the weld, and features of the molten pool; setting a normal feature range, comparing the image data features with the normal feature range, and obtaining abnormal multimodal welding data; Specifically, the method for extracting the overall features of the welding area is as follows: using the Canny algorithm to identify the contour of the workpiece, and determining the position of the workpiece in the welding area through contour matching; using the SIFT algorithm to extract the feature points on the workpiece, combining the three-dimensional model and camera calibration parameters, and calculating the posture information of the workpiece, including the rotation angle and translation vector; The method for extracting close-up features of welds is as follows: the weld area is segmented and extracted from the background by a region growing algorithm, and the width of the weld at different positions is calculated; the height information of the weld is obtained by combining the structured light measurement principle with the image data; during the welding process, light of known structure is projected onto the weld, and the weld height is calculated according to the triangulation principle by taking the deformed image of the light on the weld surface; the root mean square roughness of the weld surface is calculated, and the texture features are extracted by using the grayscale co-occurrence matrix method through the analysis of the weld surface texture; The method for extracting the molten pool features is: using a morphological processing algorithm to process the molten pool image and outline the molten pool contour; and describing the shape characteristics of the molten pool by calculating the perimeter, area and aspect ratio of the molten pool contour.

[0041] Furthermore, under the normal working state of the welding robot, the mean and standard deviation of the data in the overall characteristics of the welding area, the close-up characteristics of the weld and the molten pool characteristics are extracted, and the normal feature range is set; the image data characteristics are compared with the normal feature range, the abnormal data are marked, and the abnormal multimodal welding data is obtained.

[0042] In a specific embodiment, the contour of the workpiece is identified by the Anny algorithm for each frame of the image, and then the position of the workpiece in the welding area is determined by contour matching. A two-dimensional coordinate system is established with the upper left corner of the image as the origin, and the coordinates of the center of mass of the workpiece are recorded. As location information.

[0043] The SIFT algorithm is used to extract feature points on the workpiece, and the rotation angle of the workpiece is calculated by combining the known 3D model and camera calibration parameters. (rotation around the z-axis) and translation vector Among them, the camera internal parameter matrix , the external parameter matrix is ​​obtained through calibration.

[0044] The weld area is segmented and extracted from the background using the region growing algorithm. On the segmented weld image, the weld width is calculated every 10 pixels, and the width values ​​of 100 positions are calculated in total. .

[0045] Using the structured light measurement principle, a light with a known structure is projected onto the weld (the projected light is stripes with a stripe spacing of ), capture the deformed image of light on the weld surface. According to the triangulation principle and combined with the camera calibration parameters, the weld height h is calculated.

[0046] Calculate the RMS roughness of the weld surface First, the weld image is converted into a grayscale image, and then a 10×10 pixel window is selected in the weld area to calculate the root mean square roughness in each window. At the same time, the gray level co-occurrence matrix method is used to extract texture features and calculate characteristic values ​​such as contrast, correlation, energy and entropy.

[0047] The molten pool image is processed using morphological processing algorithms (such as erosion and expansion operations) to outline the molten pool contour. The perimeter C, area A and aspect ratio AR of the molten pool contour are calculated.

[0048] The data collected from 100 normal welding processes were statistically analyzed, the mean and standard deviation of each feature were calculated, and the normal feature range was set as follows: For the overall characteristics of the weld area: For the workpiece position: the mean x coordinate is , the standard deviation is , the normal range is ; The mean y coordinate is , the standard deviation is , the normal range is .

[0049] For workpiece posture: rotation angle The mean is The standard deviation is The normal range is degrees; translation vector The mean is , the standard deviation is , the normal range is ; The mean is , the standard deviation is , the normal range is ; The mean is , the standard deviation is , the normal range is .

[0050] For the weld close-up feature: For weld width: average width at each position The range is , the standard deviation is , the normal range is .

[0051] For weld height: the average height is , the standard deviation is , the normal range is .

[0052] For texture features: root mean square roughness The mean is , the standard deviation is , the normal range is The mean contrast is , the standard deviation is , the normal range is The mean correlation is , the standard deviation is , the normal range is The mean energy is , the standard deviation is , the normal range is The mean entropy is , the standard deviation is , the normal range is .

[0053] For the melt pool feature: For the circumference of the molten pool: the average circumference is Pixels, standard deviation is Pixels, normal range is Pixel.

[0054] For the molten pool area: the average area is pixels², with a standard deviation of pixels², the normal range is Pixel².

[0055] For the aspect ratio of the molten pool: the average aspect ratio is , the standard deviation is , the normal range is .

[0056] In the subsequent welding process, 20 welding process image data were collected for abnormality detection. Each acquisition was also 10 seconds, with a total of 600 frames of images. The above features were extracted from each frame of the image and compared with the set normal feature range.

[0057] For the overall characteristics of the welding area: During the fifth welding process, it was found that the x-coordinate of the workpiece was 950 in a certain frame, which was beyond the normal range. The frame image was marked as abnormal data in the x-coordinate of the workpiece position; during the 12th welding process, the workpiece rotation angle If it reaches 5 degrees, which is beyond the normal range, the frame image is marked as abnormal data in terms of the workpiece posture rotation angle.

[0058] For the close-up features of the weld: During the 8th welding process, the weld width at a certain position was 4.2mm, which was beyond the normal range, and the weld width at that position in the frame image was marked as abnormal data; during the 15th welding process, the weld height was 2.2mm, which was below the normal range, and the weld height in the frame image was marked as abnormal data; during the 18th welding process, the root mean square roughness was The value is 0.16, which is beyond the normal range, marking the frame image as abnormal data in terms of RMS roughness.

[0059] For the molten pool features: in the third welding process, the circumference of the molten pool is 75 pixels, which is lower than the normal range, and the frame image is marked as abnormal data in terms of the circumference of the molten pool; in the tenth welding process, the area of ​​the molten pool is 1000 pixels², which is beyond the normal range, and the frame image is marked as abnormal data in terms of the area of ​​the molten pool; in the sixteenth welding process, the aspect ratio of the molten pool is 1.5, which is beyond the normal range, and the frame image is marked as abnormal data in terms of the aspect ratio of the molten pool.

[0060] Step S400: Based on the welding data anomaly classification set and in combination with the corresponding environmental data, a weighted directed graph is used to determine the environment-welding association relationship, specifically the association relationship between different factors in the environmental data and different anomaly categories; based on the environment-welding association relationship, anomalies caused by environmental factors in the abnormal multimodal welding data are eliminated.

[0061] Specifically, nodes are created for ambient temperature, humidity and light intensity respectively, and environmental factor nodes are determined; nodes are created for each abnormal category in the welding data abnormal classification set, and abnormal category nodes are determined; for each welding abnormal category, the corresponding environmental data before or when the abnormality occurs are counted; conditional probability is used to calculate the weight of the edge; When an environmental factor is associated with an abnormal category, a directed edge is drawn from the environmental factor node to the corresponding abnormal category node, wherein the method for determining whether there is an association is as follows: if the calculated weight is not 1, it indicates that there is a non-random association between the two; and the environment-welding association is determined.

[0062] Furthermore, for the direct connection edge from the environmental factor node to the abnormal category node in the weighted directed graph, when the weight of an edge is greater than 1, it indicates that the environmental factor has a direct promoting effect on the abnormal category; when the weight of an edge is less than 1, it indicates that the environmental factor has a negative direct correlation with the abnormal category; For environmental factor nodes and abnormal category nodes that have no direct connection but indirect paths, the intermediate nodes and edge weights on the path are analyzed to determine the association relationship; By analyzing all the associated paths and weights in the weighted directed graph, combined with the welding anomalies caused by the mutual influence of environmental factors, the weights of direct and indirect associations are normalized, and the weights of all environmental factors related to a certain anomaly category are converted to the same scale. Different weight coefficients are assigned according to the importance of each factor, and the comprehensive score is calculated; According to historical data, a threshold for the comprehensive impact of environmental factors is set for each abnormal category, and the comprehensive scores are compared to eliminate the anomalies caused by environmental factors in the abnormal multimodal welding data.

[0063] In a specific embodiment, a welding production line was monitored for a period of time, and a welding data anomaly classification set and corresponding environmental data were collected. The welding data anomaly classification set contains three anomaly categories: porosity anomaly, crack anomaly and unfusion anomaly. Environmental data includes ambient temperature, humidity and light intensity. A total of 1,000 welding sample data were collected, including 700 normal samples and 300 abnormal samples. The specific abnormal sample distribution is: 100 porosity anomalies, 100 crack anomalies, and 100 unfusion anomalies. According to the calculated edge weights, since all edge weights are not 1, directed edges are drawn from each environmental factor node to the corresponding abnormal category node to form a weighted directed graph.

[0064] For pore anomaly, the edge weight from "temperature node" to "pore anomaly node" , indicating that the ambient temperature above 30℃ has a direct promoting effect on stomatal anomaly; the edge weight from "humidity node" to "stomatal anomaly node" , indicating that humidity above 60% has a direct promoting effect on stomatal abnormality; the edge weight from "light intensity node" to "stomatal abnormality node" , indicating that light intensity below 50lux has a direct promoting effect on stomatal abnormalities.

[0065] For crack anomalies, the edge weight from "temperature node" to "crack anomaly node" , indicating that the ambient temperature below 10℃ has a direct promoting effect on crack anomaly; the edge weight from "humidity node" to "crack anomaly node" , indicating that humidity below 30% has a direct promoting effect on crack anomaly; the edge weight from "light intensity node" to "crack anomaly node" , indicating that light intensity above 200 lux has a negative direct correlation with crack anomalies.

[0066] For the unfused anomaly, the edge weight from the "temperature node" to the "unfused anomaly node" , indicating that the ambient temperature between 20-25℃ has a direct promoting effect on the unfused anomaly; the edge weight from "humidity node" to "unfused anomaly node" , indicating that humidity between 40-50% has a direct promoting effect on the unfused anomaly; the edge weight from "light intensity node" to "unfused anomaly node" , indicating that light intensity between 80-120 lux has a negative direct correlation with the unfusion anomaly.

[0067] There is an indirect path: "temperature node" - "metal thermal expansion coefficient change node" - "crack abnormality node". The edge weight from "temperature node" to "metal thermal expansion coefficient change node" is 1.5, and the edge weight from "metal thermal expansion coefficient change node" to "crack abnormality node" is 1.2. The comprehensive weight of this indirect path is , indicating that temperature indirectly promotes crack anomalies by affecting the change of metal thermal expansion coefficient.

[0068] The weights of direct and indirect associations are normalized using the maximum-minimum normalization method. Taking stomatal anomaly as an example, the weights of direct association are , , The normalized weights are , , .

[0069] According to experience, the weight coefficients assigned to temperature, humidity, and light intensity are 0.5, 0.3, and 0.2, respectively. .

[0070] Similarly, calculate the comprehensive score of crack anomaly and lack of fusion anomaly. , comprehensive score of unfusion anomaly .

[0071] Based on historical data, the threshold for the comprehensive impact of environmental factors is set at 0.8 for pore anomaly, 0.75 for crack anomaly, and 0.65 for lack of fusion anomaly. , , , it can be judged that the currently detected porosity anomalies, crack anomalies and lack of fusion anomalies are all caused by environmental factors, and these anomalies are excluded from the abnormal multimodal welding data.

[0072] like Figure 2 The system structure diagram of the intelligent image data recognition and analysis system based on the AI ​​model. The present invention provides an intelligent image data recognition and analysis system based on the AI ​​model, including: Data acquisition and preprocessing module: including: data acquisition unit and real-time multi-modal welding data construction unit; wherein, the data acquisition unit obtains the image data, sound data, physical parameter data and environmental data of the welding robot during the welding process, and the real-time multi-modal welding data construction unit performs time synchronization and data cleaning, and integrates to form real-time multi-modal welding data; Abnormal classification model training and application module: including: deep neural network training unit and real-time data abnormal classification unit; wherein, the deep neural network training unit collects historical sound data and historical physical parameter data, uses deep neural network for training, and performs abnormal classification; the real-time data abnormal classification unit inputs sound data and physical parameter data into the deep neural network based on real-time multimodal welding data, and obtains the welding data abnormal classification set; Abnormal identification module: including: image data feature extraction unit, normal feature range setting unit and image data abnormal identification unit; wherein the image data feature extraction unit extracts image data features, including overall features of the welding area, close-up features of the weld and molten pool features; the normal feature range setting unit sets the normal feature range, and the image data abnormal identification unit compares the image data features with the normal feature range to obtain abnormal multi-modal welding data; Abnormality elimination module: including: a weighted directed graph construction unit and an environmental factor abnormality elimination unit; wherein, the weighted directed graph construction unit is based on the welding data abnormality classification set, combined with the corresponding environmental data, and uses a weighted directed graph to determine the environment-welding association relationship, specifically the association relationship between different factors in the environmental data and different abnormal categories; the environmental factor abnormality elimination unit eliminates abnormalities caused by environmental factors in abnormal multimodal welding data based on the environment-welding association relationship.

[0073] like Figure 3 As shown in the device line drawing of the intelligent image data recognition and analysis device based on the AI ​​model of the present invention, the present invention provides an intelligent image data recognition and analysis device based on the AI ​​model.

[0074] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. An intelligent image data recognition and analysis method based on an AI model, characterized in that: The following steps are involved: Acquire the image data, sound data, physical parameter data and environmental data of the welding robot during the welding process, perform time synchronization and data cleaning, and integrate them into real-time multi-modal welding data; Collect historical sound data and historical physical parameter data, use deep neural network for training, and perform anomaly classification; Based on real-time multimodal welding data, the sound data and physical parameter data are input into the deep neural network to obtain the abnormal classification set of welding data; Extract image data features, including overall features of the welding area, close-up features of the weld, and molten pool features; Set the normal feature range, compare the image data features with the normal feature range, and obtain abnormal multi-modal welding data; Based on the abnormal classification set of welding data and the corresponding environmental data, a weighted directed graph is used to determine the environment-welding association relationship, specifically the association relationship between different factors in the environmental data and different abnormal categories; based on the environment-welding association relationship, anomalies caused by environmental factors in abnormal multimodal welding data are eliminated.

2. The intelligent image data recognition and analysis method based on the AI ​​model according to claim 1 is characterized in that: The method of acquiring the image data, sound data, physical parameter data and environmental data of the welding robot during the welding process, performing time synchronization and data cleaning, and integrating to form real-time multi-modal welding data includes: Use an industrial camera to capture the entire welding area, close-ups of the weld, and the molten pool to obtain image data; use a microphone to collect sound signals generated during the welding process to obtain sound data; Use current sensors and voltage sensors to measure welding current and voltage respectively, use speed sensors to monitor welding speed and wire feeding speed, use joint angle sensors to obtain angle information of each joint of the welding robot, use position sensors to determine the position coordinates of the robot, and use temperature sensors to measure the temperature of each component of the welding equipment; integrate to obtain physical parameter data; Use temperature sensors and humidity sensors to obtain the ambient temperature and humidity of the welding area, and use light sensors to obtain the light intensity of the welding area; integrate and obtain environmental data; Perform time synchronization and data cleaning to integrate and form real-time multimodal welding data.

3. The intelligent image data recognition and analysis method based on AI model according to claim 1 is characterized in that: The collecting of historical sound data and historical physical parameter data, training using a deep neural network, and performing abnormal classification include: Label the historical sound data and historical physical parameter data, and divide the data into normal samples and abnormal samples; for abnormal samples, further divide them into different types of abnormalities, including welding equipment-related abnormalities, welding process-related abnormalities, and welding quality-related abnormalities; A deep neural network using multimodal fusion includes an input layer, a feature extraction layer, a fusion layer and a classification layer; in the input layer, a sound data input layer and a physical parameter data input layer are respectively set, the sound data input layer receives a normalized sound feature vector, and the physical parameter data input layer receives normalized physical parameter data; in the feature extraction layer, a convolutional neural network is used to extract features for sound data, and a fully connected layer is used to perform preliminary feature transformation for physical parameter data; in the fusion layer, the sound feature vector extracted by the convolutional neural network and the physical parameter feature vector output by the fully connected layer are concatenated to obtain a fused feature vector; in the classification layer, a fully connected layer and a Softmax activation function are used to perform abnormal classification; Perform data partitioning, train deep neural networks, adjust hyperparameters, and evaluate and optimize models; The historical sound data and the historical physical parameter data are integrated and divided into training set, validation set and test set; the cross entropy loss function is defined, and Adam is used as the optimization algorithm to update the parameters of the deep neural network; during the training process, the training data is input into the deep neural network in batches, the value of the loss function is calculated, and the parameters of the deep neural network are updated; grid search is used to evaluate the performance under different hyperparameter combinations on the validation set and select the optimal hyperparameter combination; Use the test set to evaluate the trained deep neural network, calculate the classification accuracy, recall rate and F1 value, and optimize the deep neural network.

4. The intelligent image data recognition and analysis method based on the AI ​​model according to claim 1 is characterized in that: Based on the real-time multimodal welding data, the sound data and physical parameter data are input into the deep neural network to obtain the welding data abnormal classification set, including: Based on real-time multimodal welding data, sound data and physical parameter data are input into the deep neural network. According to the probability output by the Softmax function, a threshold is set, and the category with a probability value greater than the threshold is judged as the category to which the sample belongs. The abnormal classification results are summarized to generate a welding data abnormal classification set, which includes abnormal welding data at different time points.

5. The intelligent image data recognition and analysis method based on AI model according to claim 1 is characterized in that: The extracted image data features include overall features of the welding area, close-up features of the weld and features of the molten pool, including: The method for extracting the overall features of the welding area is as follows: using the Canny algorithm to identify the workpiece contour, and determining the position of the workpiece in the welding area through contour matching; using the SIFT algorithm to extract the feature points on the workpiece, combining the three-dimensional model and camera calibration parameters, and calculating the workpiece posture information, including the rotation angle and translation vector; The method for extracting close-up features of welds is as follows: the weld area is segmented and extracted from the background by a region growing algorithm, and the width of the weld at different positions is calculated; the height information of the weld is obtained by combining the structured light measurement principle with the image data; during the welding process, light of known structure is projected onto the weld, and the weld height is calculated according to the triangulation principle by taking the deformed image of the light on the weld surface; the root mean square roughness of the weld surface is calculated, and the texture features are extracted by using the grayscale co-occurrence matrix method through the analysis of the weld surface texture; The method for extracting the molten pool features is: using a morphological processing algorithm to process the molten pool image and outline the molten pool contour; and describing the shape characteristics of the molten pool by calculating the perimeter, area and aspect ratio of the molten pool contour.

6. The intelligent image data recognition and analysis method based on AI model according to claim 1 is characterized in that: The setting of the normal feature range, comparing the image data features with the normal feature range, and obtaining abnormal multi-modal welding data includes: When the welding robot is in normal working condition, the mean and standard deviation of the data in the overall features of the welding area, close-up features of the weld and molten pool features are extracted, and the normal feature range is set; the image data features are compared with the normal feature range, the abnormal data are marked, and the abnormal multimodal welding data is obtained.

7. The intelligent image data recognition and analysis method based on AI model according to claim 1 is characterized in that: The method is based on the abnormal classification set of welding data and combined with the corresponding environmental data to determine the environment-welding association relationship using a weighted directed graph, specifically the association relationship between different factors in the environmental data and different abnormal categories, including: Create nodes for ambient temperature, humidity and light intensity respectively, and determine the environmental factor nodes; create nodes for each abnormal category in the welding data abnormal classification set, and determine the abnormal category nodes; for each welding abnormal category, count the corresponding environmental data before or when the abnormality occurs; use conditional probability to calculate the weight of the edge; When an environmental factor is associated with an abnormal category, a directed edge is drawn from the environmental factor node to the corresponding abnormal category node, wherein the method for determining whether there is an association is as follows: if the calculated weight is not 1, it indicates that there is a non-random association between the two; and the environment-welding association is determined.

8. The intelligent image data recognition and analysis method based on AI model according to claim 1 is characterized in that: The method of eliminating abnormalities caused by environmental factors in abnormal multimodal welding data based on the environment-welding association relationship includes: For the direct connection edge from the environmental factor node to the abnormal category node in the weighted directed graph, when the weight of an edge is greater than 1, it indicates that the environmental factor has a direct promoting effect on the abnormal category; when the weight of an edge is less than 1, it indicates that the environmental factor has a negative direct correlation with the abnormal category; For environmental factor nodes and abnormal category nodes that have no direct connection but indirect paths, the intermediate nodes and edge weights on the path are analyzed to determine the association relationship; By analyzing all the associated paths and weights in the weighted directed graph, combined with the welding anomalies caused by the mutual influence of environmental factors, the weights of direct and indirect associations are normalized, and the weights of all environmental factors related to a certain anomaly category are converted to the same scale. Different weight coefficients are assigned according to the importance of each factor, and the comprehensive score is calculated; According to historical data, a threshold for the comprehensive impact of environmental factors is set for each abnormal category, and the comprehensive scores are compared to eliminate the anomalies caused by environmental factors in the abnormal multimodal welding data.

9. An intelligent image data recognition and analysis system based on an AI model, using the intelligent image data recognition and analysis method based on an AI model according to any one of claims 1 to 8, characterized in that: include: Data acquisition and preprocessing module: including: data acquisition unit and real-time multi-modal welding data construction unit; wherein, the data acquisition unit obtains the image data, sound data, physical parameter data and environmental data of the welding robot during the welding process, and the real-time multi-modal welding data construction unit performs time synchronization and data cleaning, and integrates to form real-time multi-modal welding data; Abnormal classification model training and application module: including: deep neural network training unit and real-time data abnormal classification unit; wherein, the deep neural network training unit collects historical sound data and historical physical parameter data, uses deep neural network for training, and performs abnormal classification; the real-time data abnormal classification unit inputs sound data and physical parameter data into the deep neural network based on real-time multimodal welding data, and obtains the welding data abnormal classification set; Abnormal identification module: including: image data feature extraction unit, normal feature range setting unit and image data abnormal identification unit; wherein the image data feature extraction unit extracts image data features, including overall features of the welding area, close-up features of the weld and molten pool features; the normal feature range setting unit sets the normal feature range, and the image data abnormal identification unit compares the image data features with the normal feature range to obtain abnormal multi-modal welding data; Abnormality elimination module: including: a weighted directed graph construction unit and an environmental factor abnormality elimination unit; wherein, the weighted directed graph construction unit is based on the welding data abnormality classification set, combined with the corresponding environmental data, and uses a weighted directed graph to determine the environment-welding association relationship, specifically the association relationship between different factors in the environmental data and different abnormal categories; the environmental factor abnormality elimination unit eliminates abnormalities caused by environmental factors in abnormal multimodal welding data based on the environment-welding association relationship.

10. An intelligent image data recognition and analysis device based on an AI model, characterized in that: The device includes the intelligent image data recognition and analysis system based on the AI ​​model as described in claim 9 and an embedded computer.

Images