Processing abnormity monitoring method, electronic equipment and storage medium

Abstract

The invention provides a processing abnormity monitoring method, electronic equipment and a storage medium, and the method comprises the steps: obtaining a processing video obtained by shooting when a production device processes a product through a cladding material, and the processing video comprises a plurality of video frames; using a first model to classify the plurality of video frames, and determining the category of each video frame, the categories including a jitter category and a non-jitter category; if the video frames divided into the jitter category exist, determining the jitter level of the product in the processing process based on the video frame number of the video frames divided into the jitter category; based on the jitter level, obtaining a first detection result of the product in the processing process; or if there is no video frame divided into the jitter category, obtaining a first detection result of the product in the processing process based on the non-jitter category. According to the method, the monitoring cost of the product in the machining process can be reduced, and the monitoring precision is improved.

Description

Technical Field

[0001] The present application relates to the field of production and manufacturing technology, and in particular to a processing abnormality monitoring method, electronic equipment and storage medium. Background Art

[0002] Laser cladding technology is a surface improvement or repair technology. Due to its good welding performance, corrosion resistance and durability, it is widely used in aerospace, automobile manufacturing, military industry, electronic devices, medical equipment and other industries. Laser cladding technology mainly uses a laser beam to accelerate the heating and melting of the cladding material (for example, titanium wire) and the thin layer on the surface of the substrate, so that the cladding material and the substrate are melted together to form a molten pool. Subsequently, the parameters of the laser beam or laser cladding machine are adjusted so that the molten pool cools and solidifies rapidly, thereby forming a metallurgical bonding layer of a certain thickness on the surface of the substrate.

[0003] If the laser cladding process is disturbed, problems such as bubbles and voids are likely to occur, affecting product quality. At present, in order to conduct quality inspections on products processed by laser cladding technology, additional data acquisition equipment is usually required. For example, sensors are used to collect signals generated by the processing process such as sound, light, and potential difference, and quality inspection is achieved by analyzing the signals. This type of method is costly and has certain limitations. Summary of the invention

[0004] The embodiments of the present application disclose a processing abnormality monitoring method, an electronic device and a storage medium, which solve the technical problem of high product monitoring costs when products are processed using laser cladding technology.

[0005] The present application provides a processing abnormality monitoring method, the method comprising: obtaining a processing video shot when a production device processes a product using a cladding material, the processing video comprising multiple video frames; classifying the multiple video frames using a first model to determine a category of each video frame, the category comprising a jitter category and a non-jitter category; if there are video frames classified into the jitter category, based on the number of video frames classified into the jitter category, determining the jitter level of the product during the processing; based on the jitter level, obtaining a first detection result of the product during the processing; or, if there are no video frames classified into the jitter category, based on the non-jitter category, obtaining a first detection result of the product during the processing.

[0006] In some optional embodiments, the jitter level of the product during the processing process is determined based on the number of video frames classified into the jitter category, including: if there are a continuous preset number of video frames classified into the jitter category, determining the jitter level to be the first level; if there is any video frame classified into the jitter category, and there is no continuous preset number of video frames classified into the jitter category, determining the jitter level to be the second level, and the abnormality of the first level is greater than that of the second level.

[0007] In some optional embodiments, the jitter level of the product during the processing process is determined based on the number of video frames classified into the jitter category, including: if there are video frames that are continuously classified into the abnormal category within a first time length, the jitter level is determined to be the first level; if there are video frames that are continuously classified into the abnormal category within a second time length, and there are no video frames that are classified into the abnormal category beyond the second time length, the jitter level is determined to be the second level, the second time length is less than the first time length, and the degree of abnormality of the first level is greater than the second level.

[0008] In some optional embodiments, the method further includes: if there are no video frames classified into the jitter category, it is determined that the multiple video frames belong to the non-jitter category, and the first detection result is that no processing abnormalities occurred during the processing of the product; if there are video frames classified into the jitter category, then the first detection result includes processing abnormalities occurring during the processing of the product and the jitter level.

[0009] In some optional embodiments, each video frame displays the cladding material and a molten pool formed on the product after the cladding material is irradiated by a laser beam emitted by the production equipment, and the method further includes: using a second model to determine a target area in each video frame, the target area being an area where a cladding end of the cladding material close to the molten pool is located; determining a center point of the cladding end based on the area coordinates of the target area, and determining an amplitude value of the movement based on the center point; determining a second detection result based on the amplitude value; and obtaining a target detection result of the product during processing based on the first detection result and the second detection result.

[0010] In some optional embodiments, determining the center point of the cladding end based on the regional coordinates of the target area includes: determining a processing area according to a second model, and establishing a coordinate system based on the processing area; determining the regional coordinates of the target area of ​​each video frame on the coordinate system; and calculating the center point corresponding to the cladding end in each video frame according to the regional coordinates.

[0011] In some optional embodiments, determining the amplitude value of the movement based on the center point includes: calculating an average value of the center point according to the center point corresponding to the cladding end in each video frame; calculating the distance between the center point corresponding to the cladding end in each video frame and the average value to obtain the amplitude value.

[0012] In some optional embodiments, determining the second detection result based on the amplitude value includes: if the amplitude value is greater than a preset threshold, determining that the second detection result is that a processing abnormality occurs during the processing of the product; if the amplitude value is less than or equal to the preset threshold, determining that the second detection result is that no processing abnormality occurs during the processing of the product.

[0013] In some optional embodiments, the target detection result of the product during the processing is obtained based on the first detection result and the second detection result, including: if the first detection result and the second detection result both indicate that a processing abnormality occurs during the processing of the product, then the target detection result is determined to be a processing abnormality, and a first-level alarm is issued; if the first detection result or the second detection result indicates that a processing abnormality occurs during the processing of the product, then the target detection result is determined to be a potential abnormality, and a second-level alarm is issued; if the first detection result and the second detection result both indicate that no processing abnormality occurs during the processing of the product, then the target detection result is determined to be normal processing.

[0014] In some optional embodiments, the method also includes training the first model, including: obtaining a historical video taken when the production equipment uses the cladding material to process a historical product, and using the video frames in the historical video as a data set, wherein the data set includes multiple images; labeling each image in the data set with a category, wherein the category includes a jitter category and a non-jitter category; dividing the category-labeled data set into a training set and a verification set; using the training set to perform multiple iterative training on the first model, and calculating the loss value of the first model after each iterative training; adjusting the model parameters of the first model based on the loss value, and verifying the first model using the verification set to obtain an evaluation result corresponding to at least one evaluation indicator; if the evaluation result meets the preset conditions, the training of the first model is completed.

[0015] The present application also provides an electronic device, which includes a processor and a memory, and the processor is used to implement the processing abnormality monitoring method when executing a computer program stored in the memory.

[0016] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the processing abnormality monitoring method is implemented.

[0017] In the processing abnormality monitoring method provided in the present application, in order to monitor the abnormal situation during the process of the production equipment using the cladding material to process the product, the processing video obtained by shooting when the production equipment uses the cladding material to process the product is obtained, and the processing video includes multiple video frames. In order to reduce the monitoring cost, the first model is used to classify the multiple video frames to divide each video frame into a jitter category or a non-jitter category. If there are video frames classified as jitter categories, it means that an abnormality occurs during the processing, and the degree of the abnormality can be further analyzed. Based on the number of video frames classified as jitter categories, the jitter level of the product during the processing is determined, and the jitter level is judged by the number of video frames, which can reduce the complexity of the calculation and improve the accuracy of determining the degree of abnormality. Finally, based on the jitter level, the first detection result of the product during the processing is obtained, which can reduce the product monitoring cost and improve the monitoring accuracy of abnormal situations during the processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is an application environment architecture diagram of the processing anomaly monitoring method provided in an embodiment of the present application.

[0019] Figure 2 It is a schematic diagram of the use of the production equipment provided in the embodiment of the present application.

[0020] Figure 3 It is a flow chart of the processing abnormality monitoring method provided in an embodiment of the present application.

[0021] Figure 4 It is a schematic diagram of classifying video frames in the processing process provided by an embodiment of the present application.

[0022] Figure 5 It is a schematic diagram of a video frame provided by an embodiment of the present application.

[0023] Figure 6 It is a schematic diagram of a video frame provided by another embodiment of the present application.

[0024] Figure 7 It is a schematic diagram of processing abnormality monitoring provided in one embodiment of the present application.

[0025] Figure 8 This is a training flowchart of the first model provided in an embodiment of the present application.

[0026] Fig. 9 This is a training diagram of the first model provided in one embodiment of the present application.

[0027] Fig.10 This is a flow chart of a processing abnormality monitoring method provided by another embodiment of the present application.

[0028] Fig.11 It is a schematic diagram of a video frame provided by an embodiment of the present application.

[0029] Fig.12 It is a schematic diagram of a video frame provided by yet another embodiment of the present application.

[0030] Fig.13 It is a schematic diagram of the coordinate axis provided in the embodiment of the present application.

[0031] Fig.14 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] To facilitate understanding, some illustrations of concepts related to the embodiments of the present application are given by way of example for reference.

[0033] It should be noted that in this application, "at least one" means one or more, and "more than one" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and drawings of this application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0034] In order to solve the technical problem of high product monitoring costs when using laser cladding technology to process products. The embodiment of the present application provides a processing abnormality monitoring method, electronic equipment and storage medium, which does not require additional sensors and reduces the cost of collecting monitoring data. At the same time, when it is determined that an abnormality has occurred, the jitter level is distinguished, which can improve the abnormal diagnosis efficiency of subsequent abnormal cause analysis, and can optimize maintenance strategies and warn of potential abnormalities through jitter levels. The following first describes the application scenario of the processing abnormality monitoring method of the present application.

[0035] Figure 1It is an application environment architecture diagram of the processing anomaly monitoring method provided in the embodiment of the present application. The processing anomaly monitoring method provided in the embodiment of the present application is applied to an electronic device 10, and the electronic device 10 is connected to the production equipment 20 for communication. The communication connection mode includes wired network communication and wireless network communication. Among them, the wired network can be any one of a local area network, a metropolitan area network and a wide area network, and the wireless network can be any one of Bluetooth Technology, wireless local area network (Wireless Fidelity, Wi-Fi), near field communication (Near Field Communication, NFC), self-organizing network wireless communication (ZigBee Wireless Networks, ZigBee) technology, infrared data organization (Infrared Data Association, IrDA) technology, ultra-wideband (Ultra Wideband, UWB) technology, wireless universal serial bus (Universal Serial Bus, USB) and the like.

[0036] The electronic device 10 may be a computer device, a mobile phone, a tablet computer, a personal digital assistant (PDA), a cloud server, a physical server, etc. The embodiment of the present application does not impose any limitation on the specific type of the electronic device.

[0037] The production equipment 20 may be a 3D (Dimension) welding wire laser cladding welding machine. Figure 2 As shown, the production equipment 20 includes a camera device 210 and a wire feeding head device 220 .

[0038] The camera device 210 may be a charge coupled device (CCD) camera, which is used to monitor the process of processing the product M using the cladding material N to obtain a processing video. The cladding material N may be a titanium wire, and the product M may be an alloy material or a composite material containing aluminum and titanium, or other metal materials, alloy materials or composite materials, which are not limited in this application.

[0039] The wire feeding head device 220 may be a carrier for carrying the cladding material N.

[0040] In some embodiments of the present application, a cladding material N is placed in the wire feeder device 220 of the production equipment 20, and a product M to be processed is placed in the production equipment 20. During the processing, the cladding material N is irradiated by the laser beam emitted by the production equipment 20 to form a molten pool on the product M. The processing video during the processing is captured by the camera device 210, and the range captured by the camera device 210 may include the cladding end of the cladding material N close to the molten pool and the molten pool.

[0041] The indication Figure 1 This is only an example of an application scenario and does not constitute a limitation on the application scenario of the processing anomaly monitoring method provided in the embodiment of the present application. The processing anomaly monitoring method provided in the embodiment of the present application can also be applied to a production device 20 including a display screen, and a user can view the result of the processing anomaly detection performed by the production device 20 on the processing video through the display screen. In addition, the production device 20 can also send the detection result to the electronic device 10, so that the user can view the detection result through the electronic device 10.

[0042] See also Figure 3 As shown, Figure 3 is a flowchart of a processing anomaly monitoring method provided in an embodiment of the present application, which is applied to electronic equipment (such as Figure 1 According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.

[0043] Step S301, obtaining a processing video captured when a production device processes a product using a cladding material.

[0044] In some embodiments of the present application, the production equipment may be a 3D wire laser cladding welding machine, the cladding material may be titanium wire, and the product may be a component comprising a composite material.

[0045] In one example, the product may be a mobile phone case, and the production equipment may use the cladding material to fill or repair the holes or scratches on the mobile phone case. The product may also be components such as circuit boards and connectors, and the production equipment may use the cladding material to connect or repair the components to ensure the stability and reliability of the components. The product may also be a metal part, and the production equipment may use the cladding material to seamlessly connect and repair the frames, buttons, etc. of the metal parts to improve the quality and durability of the metal parts. The above are just examples, and this application does not limit the type of product.

[0046] During the processing, the production equipment emits a laser beam to melt the cladding material, which is formed in the vertical direction of the product surface or inner cavity to form a molten pool, thereby achieving the purpose of repairing the product. The camera device of the production equipment is used to shoot the cladding material and the area where the molten pool is located during the processing to obtain a processing video, which includes multiple video frames. The camera device can be a CCD camera with advantages such as high sensitivity and vibration resistance.

[0047] In the above embodiment, the camera device can observe the process of the melted cladding material forming a molten pool in the inner cavity of the product, and can promptly detect abnormalities in the inner cavity of the product during the processing, such as cracks, bubbles, impurities, etc. in the molten pool. In addition, the production equipment may experience abnormal jitter during the processing, or jitter caused by external interference. The processing video shot by the camera device can capture the position of the cladding material and provide accurate video frames for subsequent jitter analysis.

[0048] Step S302: Classify the multiple video frames using the first model to determine the category of each video frame.

[0049] In some embodiments of the present application, the first model may be a classification model, which may be deployed in an electronic device. The classification model may include any one or more combinations of models such as residual networks, long short-term memory networks (LSTM), recurrent neural networks (RNN), and convolutional neural networks (CNN). The present application does not limit the type of classification model. The training of the classification model can refer to the following Figure 8 The embodiment shown.

[0050] In some embodiments of the present application, the categories of video frames include jitter categories and non-jitter categories. After the production equipment captures the processed video through the camera device, the processed video can be uploaded to the electronic device through wired communication or wireless communication. After the electronic device receives the processed video, it calls the pre-deployed first model to classify each video frame in the processed video, thereby obtaining the category of each video frame. Figure 4 Provide explanation.

[0051] like Figure 4As shown in FIG. 1 , during the processing, the laser beam irradiates the cladding material N, and the camera device 210 captures the processing video. After receiving the processing video, the electronic device divides each video frame into a shaking category or a non-shaking category. In order to better distinguish the video frames of the shaking category from the non-shaking category, as shown in FIG. Figure 4 The video frame corresponding to the jitter category shown contains the cladding material N at two positions and the moving track of the cladding material N. The cladding material N at these two positions indicates that the position of the cladding material N has shifted during the processing. The schematic diagram of the video frame corresponding to the jitter category is only to distinguish it from the schematic diagram of the video frame of the non-jitter category. The video frame shot in the actual production process only contains the cladding material N at one position. The schematic diagrams of the video frames of the jitter category mentioned in the following embodiments are all displayed in this way and will not be repeated.

[0052] Step S303, if there are video frames classified as jitter categories, determine the jitter level of the product during processing based on the number of video frames classified as jitter categories; based on the jitter level, obtain a first detection result of the product during processing.

[0053] In some embodiments of the present application, after obtaining the category of each video frame using the first model, each video frame can be quantitatively analyzed based on the category to determine whether there are video frames classified as jitter categories, and the number of video frames classified as jitter categories can be counted. If there are video frames classified as jitter categories, it means that an abnormality has occurred in the product processing process, for example, jitter has occurred.

[0054] In some embodiments of the present application, when it is determined that jitter occurs, the jitter level can be further analyzed. Since the video frames are collected in chronological order, the number of video frames that are continuously classified as jitter categories can be counted. Based on the statistical results, if there are a continuous preset number of video frames that are classified into the jitter category, the jitter level is determined to be the first level. The continuous preset number of frames can be a number greater than N, for example, N can be set to 3. The above is only an example, and the present application does not limit the setting of the continuous preset number of frames, and it can be set according to actual conditions. The following is combined with Figure 5 Provide explanation.

[0055] like Figure 5 As shown, the processed video includes 9 video frames, and the acquisition order of the video frames is taken from left to right as an example, and the number of consecutive preset frames N is preset to 4. Based on the acquisition order, it is determined as follows Figure 5If the video frame displayed in the second row and second column in the video frame is a jitter category video frame, then count the video frames that are continuously classified as jitter categories starting from the video frame displayed in the second row and second column. Based on the statistical results, it is determined that the second row and second column, the second row and third column, the third row and first column, and the third row and second column all belong to the jitter category, and there are 4 video frames in total. Compare the statistical results and the continuous preset frame number, determine that the video frames with the continuous preset frame number are classified as the jitter category, and determine that the jitter situation is the first level.

[0056] In addition, if it is determined based on the statistical results that there are more than 4 video frames classified as the shaking category, it is also determined as a first level shaking situation.

[0057] If any video frame is classified into the jitter category, and there are no consecutive preset number of video frames classified into the jitter category, the jitter level is determined to be the second level, wherein the abnormality of the first level is greater than that of the second level, the first level may be severe jitter, and the second level may be slight jitter. Figure 6 Provide explanation.

[0058] like Figure 6 As shown, the processed video includes 9 video frames, and the acquisition order of the video frames is taken from left to right as an example, and the number of consecutive preset frames N is preset to 4. Based on the acquisition order, it is determined as follows Figure 6 If the video frame displayed in the first column of the third row in the video is a jitter category video frame, then count the video frames that are continuously classified into the jitter category starting from the video frame displayed in the first column of the third row. Based on the statistical result, it is determined that the number of video frames that are continuously classified into the jitter category is 1. Compare the statistical result and the continuous preset number of frames, determine that there is any video frame classified into the jitter category, and there is no continuous preset number of video frames classified into the jitter category, and determine that it is a second level of jitter.

[0059] In addition, if it is determined based on the statistical results that there are 2 or 3 frames that are continuously classified as jitter categories, and the continuous number is also less than the continuous preset frame number 4, it is determined to be a second level jitter situation.

[0060] In another embodiment of the present application, in addition to detecting the jitter level by presetting a continuous preset number of frames, it is also possible to determine by duration. If there are continuous video frames classified as abnormal within a first duration, the jitter level is determined to be the first level.

[0061] In one example, assuming that the first duration is 65 milliseconds, the time for determining the category of a video frame is 16 milliseconds. Figure 5For illustration, starting from the video frame (16 milliseconds) shown in the second row and second column, if all video frames classified within 65 milliseconds belong to the jitter category, it is determined that there are video frames that are continuously classified into the abnormal category within the first duration, and it is determined to be a first-level jitter situation. In addition, if there are video frames that are continuously classified into the abnormal category after exceeding the first duration, it is also determined to be a first-level jitter situation.

[0062] If there are video frames continuously classified into the abnormal category within the second time length, and there are no video frames classified into the abnormal category beyond the second time length, the jitter level is determined to be the second level, wherein the second time length is less than the first time length, and the abnormality degree of the first level is greater than the second level.

[0063] In one example, it is assumed that the second duration is 50 milliseconds. Figure 6 To explain, Figure 6 The number of video frames in the jitter category is 1, i.e. 16 milliseconds. Figure 6 The situation shown is determined as the second level. In addition, if all video frames divided within 50 milliseconds belong to the jitter category, and video frames greater than 50 milliseconds do not belong to the jitter category, it is also determined as the second level of jitter.

[0064] In some embodiments of the present application, after the occurrence of jitter is determined, the occurrence of processing abnormalities and jitter levels during the processing of the product are used as the first detection results.

[0065] Step S304: If there is no video frame classified as the shaking category, a first detection result of the product during processing is obtained based on the non-shaking category.

[0066] In some embodiments of the present application, after the category of each video frame is obtained using the first model, if there is no video frame classified as a jitter category, it means that no jitter occurs during the processing. The fact that the product does not have processing abnormalities during the processing is taken as the first detection result.

[0067] In the above embodiment, in order to monitor abnormal conditions during the process of production equipment using cladding materials to process products, a processing video obtained by shooting when the production equipment uses cladding materials to process products is obtained, and the processing video includes multiple video frames. In order to reduce the monitoring cost, the first model is used to classify the multiple video frames to divide each video frame into a jitter category or a non-jitter category. If there are video frames classified as jitter categories, it means that an abnormality occurs during the processing, and the degree of the abnormality can be further analyzed. Based on the number of video frames classified as jitter categories, the jitter level of the product during the processing is determined, and the jitter level is judged by the number of video frames, which can reduce the complexity of the calculation and improve the accuracy of determining the degree of abnormality. Finally, based on the jitter level, the first detection result of the product during the processing is obtained, which can reduce the product monitoring cost and improve the monitoring accuracy of abnormal conditions during the processing.

[0068] In some embodiments of the present application, Figure 7 As shown, the processed video is captured by the shooting device 210. Each video frame in the processed video is classified by the first model to obtain the category of each video frame. Based on the category, a plurality of video frames are quantitatively analyzed, including: counting the number of video frames classified as jitter categories and recording the duration occupied by the video frames classified as jitter categories. In the process of performing the quantitative analysis, the statistical results of the quantitative analysis, the recorded results, and the plurality of video frames of the processed video can be stored in a database of the electronic device.

[0069] After determining the first test result, the first test result can be fed back to the user or the production equipment. The feedback method includes: displaying the first test result to the user through the display screen of the electronic device, or sending the first test result to the terminal device held by the user, or feeding back the first test result to the relevant application system in the production equipment. This application does not limit the method and object of sending the first test result.

[0070] When the first detection result includes processing anomalies and jitter levels in the product processing process, the user can maintain the processing process of the production equipment and intercept the materials required for production to prevent batch defects. In addition, when the production equipment receives the first detection result including processing anomalies and jitter levels in the product processing process, it can also trigger the stop of production to prevent batch defects.

[0071] Figure 8 : is a training flow chart of the first model provided in the embodiment of the present application. Figure 3 The first model shown can be Figure 8 The training is performed by the steps in the embodiment shown. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted. Figure 8As shown, the following steps are included.

[0072] Step S801, obtaining a historical video shot when a production device processes a historical product using a cladding material, and using video frames in the historical video as a data set.

[0073] In some embodiments of the present application, the production equipment may be a 3D wire laser cladding welding machine. The cladding material may be a titanium wire. The historical product may be a composite material to be processed. In order to train the first model, a historical video taken when the production equipment processes the historical product may be obtained. Video frames in the historical video are extracted as a data set.

[0074] Step S802: label each image in the data set with a category.

[0075] In some embodiments of the present application, the first model is trained in a supervised manner, and each video frame in the data set can be labeled with a category, which includes a jitter category and a non-jitter category, which can be represented by labels 0 and 1.

[0076] Step S803: divide the data set with category annotation into a training set and a validation set.

[0077] In some embodiments of the present application, the data set is divided into a training set and a validation set according to a preset division ratio. For example, 80% of the data set with category annotations can be used as a training set, and 20% can be used as a validation set.

[0078] Step S804, performing multiple iterative training on the first model using the training set, and calculating the loss value of the first model after each iterative training.

[0079] In some embodiments of the present application, the first model may be a residual network, specifically a residual network 50 (Resnet50). Resnet50 includes a convolutional layer, a maximum pooling layer, multiple groups of residual blocks, an average pooling layer, a fully connected layer, and an output layer (output through a Softmax function). Among them, the multiple groups of residual blocks may be 4 groups of residual blocks, including a first group of residual blocks, a second group of residual blocks, a third group of residual blocks, and a fourth group of residual blocks.

[0080] The first group of residual blocks may include 3 residual blocks, each residual block consists of 3 convolutional layers, including a convolutional layer with a convolution kernel size of 1×1 and 64 convolution kernels, a convolutional layer with a convolution kernel size of 3×3 and 64 convolution kernels, and a convolutional layer with a convolution kernel size of 1×1 and 256 convolution kernels.

[0081] The second group of residual blocks may include 4 residual blocks, each residual block consists of 3 convolutional layers, including a convolutional layer with a convolution kernel size of 1×1 and 128 convolution kernels, a convolutional layer with a convolution kernel size of 3×3 and 128 convolution kernels, and a convolutional layer with a convolution kernel size of 1×1 and 512 convolution kernels.

[0082] The third group of residual blocks may include 6 residual blocks, each residual block consists of 3 convolutional layers, including a convolutional layer with a convolution kernel size of 1×1 and 256 convolution kernels, a convolutional layer with a convolution kernel size of 3×3 and 256 convolution kernels, and a convolutional layer with a convolution kernel size of 1×1 and 1024 convolution kernels.

[0083] The fourth group of residual blocks may include 3 residual blocks, each residual block consists of 3 convolutional layers, including a convolutional layer with a convolution kernel size of 1×1 and a number of convolution kernels of 512, a convolutional layer with a convolution kernel size of 3×3 and a number of convolution kernels of 512, and a convolutional layer with a convolution kernel size of 1×1 and a number of convolution kernels of 2048.

[0084] Among them, the first convolution layer of the three convolution layers contained in each group of residual blocks is used to reduce the dimension of the input features (or the number of channels), the second convolution layer is used to extract features, and the third convolution layer is used to increase (or increase) the dimension of the features.

[0085] The following combines the structure of the above residual network and takes one iteration training as an example to illustrate the training process of the first model. The first model is trained using the training set, taking the first training image (224×224) in the input training set as an example. The first training image first passes through a convolution layer (conv1) with 64 channels, a convolution kernel size of 7×7, and a stride of 2, and obtains the second training image (112×112) output by conv1, and the number of output channels of conv1 is 64. The second training image passes through a 3×3 maximum pooling layer (Max Pooling Layer) to obtain the third training image (56×56) output by the Max Pooling Layer, and the number of output channels of the Max Pooling Layer is 64. Stack 4 residual modules, and use the 4 residual modules after the stack to process the third training image to obtain the fourth training image. The fourth training image passes through the average pooling layer, the fully connected layer, and the output layer in turn to obtain the classification of the first training image by the current iterative training output, which is recorded as the predicted value.

[0086] The loss value is calculated based on the predicted value and the sample label corresponding to the training set. The BCE (Binary Cross Entropy Loss) loss function can be used for calculation, which is expressed as follows: ； In the formula, represents the predicted value of the output of the i-th iteration training, It represents the sample label corresponding to the i-th iteration training, and N is the number of images in the training set.

[0087] In the above embodiment, by using the residual structure of the residual module in the residual network, the gradient vanishing problem can be reduced and the training accuracy and effect can be improved.

[0088] Step S805: adjust the model parameters of the first model based on the loss value, verify the first model using the verification set, and obtain an evaluation result corresponding to at least one evaluation indicator.

[0089] In some embodiments of the present application, if the loss value is greater than a preset loss threshold, a gradient may be calculated, and the model parameters of the first model may be adjusted by the gradient. After adjusting the model parameters of the first model, the first model is verified using a validation set to obtain an evaluation result corresponding to at least one evaluation indicator. The evaluation indicators may include accuracy, precision, recall, etc.

[0090] Step S806: If the evaluation result meets the preset conditions, the training of the first model is completed.

[0091] In some embodiments of the present application, since the evaluation results obtained based on the validation set can better reflect the training status of the model, even if the loss function on the training set has not yet fully converged, the first model training can be considered completed. For example, when the loss value is greater than the preset loss threshold, if the evaluation result meets the preset conditions, the training of the first model is completed. In addition, in order to ensure the accuracy of the first model, the training of the first model can be completed when the loss value is less than or equal to the preset loss threshold and the evaluation result meets the preset conditions. The trained first model is stored in an electronic device so as to be executed as follows: Figure 3 The steps of the embodiment shown.

[0092] In the above embodiment, the first model is trained in a supervised manner to improve the classification accuracy of the first model during the processing.

[0093] To better understand the Figure 8 The training process shown, combined with Fig. 9 Provide explanation. Fig. 9 is a training diagram of the first model provided in the embodiment of the present application. Fig. 9As shown, the historical video is shot by the shooting device 210. Each video frame in the historical video is used as a data set for training, and each video frame in the historical video is categorized and marked as a non-abnormal category and an abnormal category. The data set is divided into a training set and a verification set according to a preset ratio. The first model is trained using the training set, and the first model is verified using the verification set, so that when the preset conditions are met, the training is completed and the trained first model is obtained. The first model is stored in the electronic device so as to be executed as follows Figure 3 Steps of the illustrated embodiment.

[0094] Fig.10 This is a flowchart of a processing abnormality monitoring method provided by another embodiment of the present application. Figure 3 The second model is introduced to determine whether the jitter condition contained in the first detection result of the first model is accurate. When the first model is combined with the second model, the accuracy of machining abnormality monitoring can be improved.

[0095] Step S1001: determine the target area in each video frame using the second model.

[0096] In some embodiments of the present application, the second model may be an open source model Yolov8. During the training process, the structure of the open source model Yolov8 mainly includes four modules, namely, Backbone module, FPN module, Head module, and Loss module. Among them, the Backbone module is used to extract features from the input image through a convolution layer, including multi-layer features. The FPN module is used to perform upsampling operations on the multi-layer features extracted by the Backbone module and perform multi-scale feature fusion and enhancement in order to improve prediction accuracy. The Head module is used to perform a series of convolution operations and full connection operations on the output features of the FPN module, and convert the input image features into prediction boxes, confidence levels, category probabilities and other results. The Loss module is used to evaluate the training of the open source model Yolov8. In addition, the loss function used by the Loss module can be a weighted loss between the detection focal loss function (Detection Focal Loss), the bounding box regression loss function (Box Regression Loss), and the classification loss function (Classification Loss). The purpose of calculating the loss function is to adjust the parameters of the model. In the training process, if the negative samples (pictures without the processing area and the cladding end) are more than the positive samples (pictures with the processing area and the cladding end), the Detection Focal Loss can solve the problem of sample imbalance, the Box Regression Loss can improve the accuracy of locating the target area, and the Classification Loss can improve the accuracy of category division.

[0097] The training data used in the training process of the above Yolov8 model includes labeled positive samples and unlabeled negative samples. Fig.11 As shown, the marked positive sample includes mark 1: processing area and mark 2: cladding end, wherein the cladding end is marked by a matrix frame.

[0098] In some embodiments of the present application, after the second model is trained, the second model is used to monitor abnormalities in the processing process. The target area in each video frame is determined by the second model, and the target area is the area where the cladding end of the cladding material is close to the molten pool, such as Fig.11 The area where the cladding end is marked during training.

[0099] Step S1002: determining the center point of the cladding end based on the area coordinates of the target area, and determining the amplitude value of the movement based on the center point.

[0100] In some embodiments of the present application, the second model can identify the processing area in addition to the target area. The center point can be the center point position of the cross section of the cladding end. In order to calculate the center point of the cladding end, a coordinate system can be established based on the processing area, and the regional coordinates of the target area of ​​each video frame on the coordinate system can be determined. The center point of the cladding end in the corresponding video frame is calculated according to each regional coordinate. Fig.12 and Fig.13 To illustrate the central point.

[0101] like Fig.12 As shown in the figure, a coordinate system is established based on the processing area to determine the regional coordinates of the target area on the coordinate system. The center point of each cladding end can be calculated through the regional coordinates. Fig.13 As shown in , mark the center point corresponding to each video frame on the coordinate system. Fig.13 (1) represents the center point of the video frame without jitter category, that is, the video frame without jitter category in the processed video. Fig.13 (2) represents the center point of the video frame with jitter category, that is, the video frame with jitter category in the processed video.

[0102] In some embodiments of the present application, the video frames obtained based on the processed video have a time sequence, the center point of each video frame is determined according to the time sequence, and the average value of the center point of each video frame is calculated, which is expressed by the following formula: (x,y) = ; In the formula, (x,y) represents the average value of the center point of the i-th video frame, n represents the number of video frames in the processed video, Represents the coordinates of the center point of the i-th video frame, where x represents the horizontal coordinate and y represents the vertical coordinate.

[0103] After determining the average value, the distance between the center point corresponding to the cladding end in each video frame and the average value can be calculated to obtain the amplitude value, which is expressed as follows: = ; Where D represents the distance.

[0104] Step S1003: determine a second detection result based on the amplitude value.

[0105] In some embodiments of the present application, since the amplitude value is calculated based on the distance between the center point of the corresponding video frame and the average value, it is possible to determine whether the video frame was shot when shaking occurred by the amplitude value. If the amplitude value is greater than a preset threshold, the second detection result is determined to be that a processing abnormality occurred during the processing of the product. If the amplitude value is less than or equal to the preset threshold, the second detection result is determined to be that no processing abnormality occurred during the processing of the product.

[0106] Step S1004, based on the first detection result and the second detection result, obtain the target detection result of the product during the processing.

[0107] In some embodiments of the present application, the Figure 3 After the first detection result is obtained in the illustrated embodiment, the target detection result can be determined in combination with the second detection result, thereby improving the accuracy of abnormality detection. If both the first detection result and the second detection result indicate that a processing abnormality occurs during the processing of the product, the target detection result is determined to be a processing abnormality, and a first-level alarm prompt is issued. If the first detection result or the second detection result indicates that a processing abnormality occurs during the processing of the product, the target detection result is determined to be a potential abnormality, and a second-level alarm prompt is issued. Among them, the methods of the first-level alarm prompt and the second-level alarm prompt are not restricted. If both the first detection result and the second detection result indicate that no processing abnormality occurs during the processing of the product, the target detection result is determined to be normal processing.

[0108] Through the above embodiment, the second model has the advantage of quickly processing video frames while ensuring detection accuracy. At the same time, through the second model, the accuracy of abnormality detection can be prompted, so that the situation of the processing process can be accurately detected.

[0109] Fig.14 Schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Figure 1 The electronic device 10 shown may include a communication module 110, a memory 120, a processor 130, an input / output (I / O) interface 140, and a bus 150. The processor 130 is coupled to the communication module 110, the memory 120, and the I / O interface 140 through the bus 150.

[0110] The communication module 110 may include a wired communication module and / or a wireless communication module. The wired communication module may provide one or more wired communication solutions such as universal serial bus (USB), controller area network bus (CAN, Controller Area Network). The wireless communication module may provide one or more wireless communication solutions such as wireless fidelity (wireless fidelity, Wi-Fi), Bluetooth (Bluetooth, BT), mobile communication network, frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR), etc.

[0111] The memory 120 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM). The random access memory can be directly read and written by the processor 130, and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, and can also be used to store user and application data. The random access memory may include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc.

[0112] The non-volatile memory may also store executable programs and user and application data, etc., and may be loaded into the random access memory in advance for direct reading and writing by the processor 110. The non-volatile memory may include a disk storage device and a flash memory.

[0113] The memory 120 is used to store one or more computer programs. The one or more computer programs are configured to be executed by the processor 130. The one or more computer programs include a plurality of instructions, and when the plurality of instructions are executed by the processor 130, the processing abnormality monitoring method executed on the server 80 can be implemented.

[0114] In other embodiments, the server 80 further includes an external memory interface for connecting to an external memory to expand the storage capacity of the server 80 .

[0115] The processor 130 may include one or more processing units, for example, the processor 130 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0116] The processor 130 provides computing and control capabilities. For example, the processor 130 is used to execute computer programs stored in the memory 120 .

[0117] The I / O interface 140 is used to provide a channel for user input or output. For example, the I / O interface 140 can be used to connect various input and output devices, such as a mouse, keyboard, touch device, display screen, etc., so that the user can enter information or visualize information.

[0118] The bus 150 is at least used to provide a channel for mutual communication among the communication module 110 , the memory 120 , the processor 130 , and the I / O interface 140 in the server 80 .

[0119] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the server 80. In other embodiments of the present application, the server 80 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0120] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above-mentioned data extraction method embodiment or each process of the above-mentioned data extraction method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. Among them, the computer-readable storage medium is, such as a read-only memory (Read-Only Memory, referred to as ROM), a random access memory (Random Access Memory, referred to as RAM), a disk or an optical disk, etc.

[0121] The embodiment of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the server reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the server executes the data extraction method described above in the embodiment of the present application.

[0122] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0123] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0124] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for monitoring machining abnormalities, characterized in that: The method comprises: Acquire a processing video captured when a production device processes a product using a cladding material, wherein the processing video includes a plurality of video frames; Classifying the plurality of video frames using the first model to determine a category of each video frame, wherein the category includes a shaking category and a non-shaking category; If there are video frames classified into the jitter category, determine the jitter level of the product during processing based on the number of video frames classified into the jitter category; obtain a first detection result of the product during processing based on the jitter level; or, If there is no video frame classified into the shaking category, a first detection result of the product during the processing is obtained based on the non-shaking category.

2. The processing abnormality monitoring method according to claim 1, characterized in that: The determining the jitter level of the product during processing based on the number of video frames classified into the jitter category includes: If there are a continuous preset number of video frames classified into the jitter category, determining the jitter level to be the first level; If any video frame is classified into the jitter category, and no consecutive preset number of video frames are classified into the jitter category, the jitter level is determined to be the second level, and the abnormality of the first level is greater than that of the second level.

3. The processing abnormality monitoring method according to claim 1, characterized in that: The determining the jitter level of the product during processing based on the number of video frames classified into the jitter category includes: If there are video frames continuously classified into the abnormal category within the first time length, determining that the jitter level is the first level; If there are video frames continuously classified into the abnormal category within the second time length, and there are no video frames classified into the abnormal category beyond the second time length, the jitter level is determined to be the second level, the second time length is less than the first time length, and the abnormality degree of the first level is greater than the second level.

4. The processing abnormality monitoring method according to claim 1, characterized in that: The method further comprises: If there is no video frame classified as the jitter category, it is determined that the multiple video frames belong to the non-jitter category, and the first detection result is that no processing abnormality occurs during the processing of the product; If there are video frames classified into the jitter category, the first detection result includes processing abnormalities occurring during the processing of the product and the jitter level.

5. The machining abnormality monitoring method according to claim 1, characterized in that: Each of the video frames displays the cladding material and a molten pool formed on the product after the cladding material is irradiated by the laser beam emitted by the production equipment, and the method further includes: Determine a target area in each of the video frames using a second model, wherein the target area is an area where a cladding end of the cladding material close to the molten pool is located; Determine the center point of the cladding end based on the area coordinates of the target area, and determine the amplitude value of the movement based on the center point; determining a second detection result based on the amplitude value; Based on the first detection result and the second detection result, a target detection result of the product during the processing is obtained.

6. The machining abnormality monitoring method according to claim 5, characterized in that: The step of determining the center point of the cladding end based on the regional coordinates of the target area includes: Determine a processing area according to the second model, and establish a coordinate system based on the processing area; Determine the region coordinates of the target region of each video frame on the coordinate system; The center point corresponding to the cladding end in each video frame is calculated according to the area coordinates.

7. The machining abnormality monitoring method according to claim 6, characterized in that: The step of determining the amplitude value of the movement based on the center point comprises: According to the center point corresponding to the cladding end in each video frame, calculating the average value of the center point; The distance between the center point corresponding to the cladding end in each video frame and the average value is calculated to obtain the amplitude value.

8. The machining abnormality monitoring method according to claim 5, characterized in that: The determining a second detection result based on the amplitude value comprises: If the amplitude value is greater than a preset threshold, determining that the second detection result is that a processing abnormality occurs during the processing of the product; If the amplitude value is less than or equal to a preset threshold value, it is determined that the second detection result is that no processing abnormality occurs during the processing of the product.

9. The machining abnormality monitoring method according to claim 5, characterized in that: The obtaining, based on the first detection result and the second detection result, a target detection result of the product during the processing, comprises: If both the first detection result and the second detection result indicate that a processing abnormality occurs during the processing of the product, the target detection result is determined to be a processing abnormality, and a first-level alarm is issued; If the first detection result or the second detection result indicates that a processing abnormality occurs during the processing of the product, the target detection result is determined to be a potential abnormality, and a second-level alarm is issued; If both the first detection result and the second detection result indicate that no processing abnormality occurs during the processing of the product, then the target detection result is determined to be normal processing.

10. The machining abnormality monitoring method according to any one of claims 1 to 9, characterized in that: The method further includes training the first model, including: Acquire a historical video taken when the production equipment uses the cladding material to process a historical product, and use video frames in the historical video as a data set, wherein the data set includes a plurality of images; Performing category labeling on each image in the data set, wherein the category includes the jitter category and the non-jitter category; Dividing the data set labeled with categories into a training set and a validation set; Performing multiple iterative training on the first model using the training set, and calculating the loss value of the first model after each iterative training; Adjusting a model parameter of the first model based on the loss value, validating the first model using the validation set, and obtaining an evaluation result corresponding to at least one evaluation indicator; If the evaluation result meets the preset conditions, the training of the first model is completed.

11. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the processor is used to execute a computer program stored in the memory to implement the machining abnormality monitoring method as claimed in any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by a processor, the processing abnormality monitoring method according to any one of claims 1 to 10 is implemented.

Images