Stamping Groove Cleaning Method and System Based on Machine Vision

Abstract

The present application proposes a stamping groove cleaning method and system based on machine vision. The stamping groove cleaning method based on machine vision includes: obtaining a monitoring video of the stamping groove in real time, where the monitoring video includes at least one monitoring image; determining multi-modal data of the stamping groove corresponding to the time stamp according to the time stamp of obtaining the monitoring image; the multi-modal data at least includes the debris weight in the stamping groove and the air flow pressure of the air compressor; performing state estimation on the stamping groove according to the monitoring image and the corresponding multi-modal data to obtain the debris state in the stamping groove; determining the output parameters of the air compressor according to the debris state; and controlling the output state of the air compressor to the stamping groove based on the output parameters. The present application relates to the technical field of intelligent manufacturing and can improve the efficiency of stamping groove cleaning.

Description

Technical Field

[0001] This application relates to the field of intelligent manufacturing technology, and particularly to a stamping groove cleaning method and system based on machine vision. Background Art

[0002] Currently, with the development of data science, some enterprises or organizations tend to improve productivity and the efficiency of production equipment maintenance through data analysis in the production environment. During the production and maintenance processes of a stamping production line, an air compressor is usually used to clean debris in the stamping groove. To improve the cleaning efficiency, a cleaning strategy is usually formulated based on digital control technology and image recognition technology. However, this method has insufficient adaptability to dynamic environments and cannot adjust the debris detection strategy according to light changes and the diversity of debris forms, easily leading to problems of false cleaning or missed detection. Moreover, when using a single modality of data to assist in cleaning, the temporal characteristics of the debris accumulation process and the environmental information in the production environment are not utilized, which may result in over-blowing (energy waste) or incomplete cleaning, and thus cause the problem of low cleaning efficiency. Summary of the Invention

[0003] In view of the above, it is necessary to propose a stamping groove cleaning method and system based on machine vision to solve the technical problems of low cleaning efficiency and low accuracy in cleaning the stamping groove.

[0004] This application provides a stamping groove cleaning method based on machine vision, which is applied to an electronic device. The electronic device is communicatively connected to an air compressor. The method includes: acquiring a monitoring video of the stamping groove in real time, where the monitoring video includes at least one monitoring image; determining multi-modal data of the stamping groove corresponding to the time stamp according to the time stamp of acquiring the monitoring image; performing state estimation on the stamping groove according to the monitoring image and the corresponding multi-modal data to obtain the debris state in the stamping groove; determining the output parameters of the air compressor according to the debris state; and controlling the output state of the air compressor to the stamping groove based on the output parameters.

[0005] In some embodiments, the performing state estimation on the stamping groove according to the monitoring image and the corresponding multi-modal data to obtain the debris state in the stamping groove includes: processing the monitoring image according to a pre-trained target recognition model to obtain attribute data of the debris in the monitoring image; determining the cumulative trend of the debris in the monitoring image corresponding to the time stamp according to the attribute data of the debris in multiple monitoring images; and predicting the debris state in the stamping groove according to the cumulative trend and the multi-modal data.

[0006] In some embodiments, the method further includes training the target recognition model, and the training of the target recognition model includes: obtaining historical images in a historical monitoring video, where the historical images correspond to label attribute data; inputting the historical images into a pre-constructed initial recognition model to obtain predicted attribute data corresponding to the historical images; determining a first loss value of the initial recognition model according to the predicted attribute data and the label attribute data; updating the initial recognition model according to the backpropagation algorithm until the first loss value meets a preset condition, and stopping updating the initial recognition model to obtain a target recognition model trained to a converged state.

[0007] In some embodiments, the determining the first loss value of the initial recognition model according to the predicted attribute data and the label attribute data includes: constructing a first loss function according to the predicted attribute data and the label attribute data, and determining the first loss value according to the first loss function, and the first loss function satisfies the following relational expression:

[0008] ; where Loss1 represents the first loss value of the initial recognition model; i is the index of the historical image, m is the number of the historical images; j is the index of the debris in the historical image indicated by the label attribute data, n is the number of the debris in the historical image indicated by the label attribute data; is the abscissa of the center point of the debris with index j in the i-th historical image indicated by the label attribute data, is the abscissa of the center point of the debris with index j in the i-th historical image indicated by the predicted attribute data; is the ordinate of the center point of the debris with index j in the i-th historical image indicated by the label attribute data, is the ordinate of the center point of the debris with index j in the i-th historical image indicated by the predicted attribute data; is the number of pixel points occupied by the debris with index j in the i-th historical image indicated by the label attribute data, is the number of pixel points occupied by the debris with index j in the i-th historical image indicated by the predicted attribute data; is the probability value that the debris with index j in the i-th historical image indicated by the label attribute data belongs to the k-th category, is the probability value that the debris with index j in the i-th historical image indicated by the predicted attribute data belongs to the k-th category; k is the index of the category corresponding to the debris, and f is the number of categories corresponding to the debris.

[0009] In some embodiments, determining the cumulative trend of debris in the monitoring images corresponding to the time stamp according to the attribute data of debris in multiple monitoring images includes: determining the moving distance and moving direction of the debris in the stamping groove corresponding to each time stamp according to the attribute data of debris in any two adjacent monitoring images; determining the cumulative trend of the debris in the monitoring images corresponding to the time stamp according to the multimodal data corresponding to each time stamp and the moving distance and moving direction of the debris in the stamping groove corresponding to each time stamp; the cumulative trend is used to indicate the cumulative weight and cumulative range corresponding to each type of debris.

[0010] In some embodiments, predicting the debris state in the stamping groove according to the cumulative trend and the multimodal data includes: determining the state data corresponding to the time stamp according to the cumulative trend and the multimodal data; predicting the debris state in the stamping groove corresponding to the next time stamp according to the state data corresponding to the time stamp and the output parameters of the air compressor corresponding to the time stamp, including:

[0011] ; wherein, represents the debris state in the stamping groove corresponding to the time stamp t + 1; represents the state data in the stamping groove corresponding to the time stamp t; represents the output parameter of the air compressor at time t, represents the weight matrix of the output parameter of the air compressor at the time stamp t; represents the error data at the time stamp t set in advance.

[0012] In some embodiments, determining the output parameter of the air compressor according to the debris state includes:

[0013] ; wherein, represents the output parameter of the air compressor at the time stamp t + 1; represents the output parameter of the air compressor at the time stamp t; represents the debris state in the stamping groove at the time stamp t - 1; represents the debris state in the stamping groove at the time stamp t; e represents the natural constant.

[0014] The embodiment of the present application also provides a stamping groove cleaning system based on machine vision. The system includes an electronic device, and the electronic device is communicatively connected to an air compressor; the electronic device is configured to obtain a monitoring video of the stamping groove in real time, and the monitoring video includes at least one monitoring image; the electronic device is further configured to determine multimodal data of the stamping groove corresponding to the timestamp according to the obtained timestamp of the monitoring image; the electronic device is further configured to perform state estimation on the stamping groove according to the monitoring image and the corresponding multimodal data to obtain the debris state in the stamping groove; the electronic device is further configured to determine the output parameters of the air compressor according to the debris state; the electronic device is further configured to control the output state of the air compressor to the stamping groove based on the output parameters.

[0015] In some embodiments, the electronic device is further configured to: process the monitoring image according to a pre-trained target recognition model to obtain attribute data of the debris in the monitoring image; determine the cumulative trend of the debris in the monitoring image corresponding to the timestamp according to the attribute data of the debris in multiple monitoring images; predict the debris state in the stamping groove according to the cumulative trend and the multimodal data.

[0016] In some embodiments, the electronic device is further configured to: obtain historical images in a historical monitoring video, and the historical images correspond to label attribute data; input the historical images into a pre-constructed initial recognition model to obtain predicted attribute data corresponding to the historical images; determine a first loss value of the initial recognition model according to the predicted attribute data and the label attribute data; update the initial recognition model according to the backpropagation algorithm until the first loss value meets a preset condition, and stop updating the initial recognition model to obtain a target recognition model trained to a convergent state.

[0017] It can be seen from the above technical solutions that the embodiment of the present application first determines the attribute data of the debris based on multiple monitoring images in the monitoring video, thereby improving the detection efficiency of the debris in the stamping groove. Then, according to the attribute data of the debris and the multimodal data obtained in real time, the debris state in the stamping groove is determined. In this way, the position change and the change of the cumulative trend of the debris in the stamping groove can be characterized by the debris state, and the correlation between the output parameters of the air compressor and the debris state in the time dimension can be determined according to the debris states corresponding to different timestamps. In this way, the influence degree of the output parameters of the air compressor on the debris state in the stamping groove can be determined in the time dimension and the space dimension. Finally, the output parameters of the air compressor are updated according to the predicted debris state. In this way, the output state of the air compressor can be adjusted based on the data related to the debris in the stamping groove obtained in real time, and further the cleaning efficiency and accuracy of the stamping groove can be improved. Description of the Drawings

[0018] Figure 1It is an application scenario diagram of a stamping groove cleaning method based on machine vision provided by an embodiment of the present application.

[0019] Figure 2 It is a flowchart of a stamping groove cleaning method based on machine vision provided by an embodiment of the present application.

[0020] Figure 3 It is a functional module diagram of a stamping groove cleaning device based on machine vision provided by an embodiment of the present application.

[0021] Figure 4 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0022] In order to more clearly understand the purpose, features, and advantages of the present application, the present application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, without conflict, the embodiments of the present application and the features in the embodiments can be combined with each other. Many specific details are set forth in the following description in order to fully understand the present application. The described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments.

[0023] In addition, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of the described features. In the description of the present application, "a plurality of" means two or more, unless otherwise specifically defined.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs. The terms used in the description of this application herein are only for the purpose of describing specific embodiments, and are not intended to limit this application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0025] The embodiment of the present application provides a stamping groove cleaning method based on machine vision, which can be applied to one or more electronic devices. An electronic device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, a microprocessor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, etc.

[0026] An electronic device can be any electronic product that can perform human-computer interaction with a customer. For example, a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), a game console, an Internet Protocol Television (IPTV), a smart wearable device, etc.

[0027] The electronic device may further include a network device and / or a client device. Among them, the network device includes, but is not limited to, a single network server, a server group composed of multiple network servers, or a cloud composed of a large number of hosts or network servers based on cloud computing.

[0028] The network where the electronic device is located includes, but is not limited to, the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (VPN), etc.

[0029] As Figure 1 shown in the application scenario diagram of a stamping groove cleaning method based on machine vision provided by an embodiment of the present application, a stamping groove cleaning method based on machine vision provided by the present application can be applied to the electronic device 100. Among them, the electronic device 100 is communicatively connected to the air compressor 200 and the imaging device 300. The electronic device 100 obtains in real time the monitoring video of the stamping groove 400 captured by the imaging device 300, and the monitoring video includes at least one monitoring image. The electronic device 100 also determines the multi-modal data of the stamping groove 400 corresponding to the time stamp according to the time stamp of obtaining the monitoring image, and performs state estimation on the stamping groove 400 according to the monitoring image and the corresponding multi-modal data to obtain the debris state in the stamping groove 400. The electronic device 100 also determines the output parameters of the air compressor 200 according to the debris state, and controls the output state of the air compressor 200 on the stamping groove 400 based on the output parameters.

[0030] As Figure 2 shown, it is a flowchart of a stamping groove cleaning method based on machine vision provided by an embodiment of the present application. According to different requirements, the order of steps in this flowchart can be changed, and some steps can be omitted. A stamping groove cleaning method based on machine vision provided by an embodiment of the present application includes the following steps.

[0031] S20, obtain the monitoring video of the stamping groove in real time, and the monitoring video includes at least one monitoring image.

[0032] In an embodiment of the present application, when cleaning the stamping groove in an industrial production environment, the monitoring video of the stamping groove can be obtained in real time, and the monitoring video includes at least one monitoring image. Subsequently, the information of the debris in the stamping groove is extracted through the video monitoring image. Among them, the debris information in the stamping groove includes multiple dimensions, which can provide a data basis for the decision of debris removal.

[0033] Exemplarily, the debris information in the stamping groove includes: geometric feature information, dynamic behavior information, physical quantity information, abnormal feature information, and spatio-temporal association information. Specifically, the geometric feature information can be a heat map of debris distribution based on kernel density analysis of density estimation, which can be used to optimize the nozzle layout and purge path planning of the air compressor; the geometric feature information can also be the accumulation morphology data of debris obtained by three-dimensional point cloud reconstruction, which can be used to predict the landslide risk of debris in the stamping groove and predict the cleaning priority of debris; the geometric feature information can also be the coverage area of debris obtained by pixel statistics after semantic segmentation, which can be used to trigger the grading alarm threshold; the geometric feature information can also be the projection contour of debris determined by the edge detection algorithm, which can be used to judge the shape characteristics of debris (for example, needle-shaped, flaky or granular). For example, long strip-shaped metal debris can be identified through contour analysis, so as to automatically adjust the blowing angle to prevent debris entanglement. The dynamic behavior information can be the movement trajectory of debris obtained by the optical flow method, which can be used to predict the source of debris generation; the dynamic behavior information can also be the splash velocity of debris obtained by the target tracking algorithm, which can be used to calculate the impact risk caused by the kinetic energy of debris on the stamping groove; the dynamic behavior information can also be the accumulation rate of debris obtained by temporal difference image analysis, which can be used to predict the maintenance cycle of the stamping groove. For example, when the horizontal velocity of debris exceeds 3.2 m / s, a predictive purge strategy can be adopted to improve the cleaning efficiency. The physical quantity information can be the equivalent weight of debris determined by the volume density integration method; the physical quantity information can also be the accumulation angle of debris calculated based on the tilt angle of the minimum bounding box; the physical quantity information can also be the particle size distribution of debris determined by the watershed algorithm; the physical quantity information can also be the surface roughness of debris obtained by texture analysis based on the gray-level co-occurrence matrix. The abnormal feature information can be the abnormal situation of spark splash of debris detected based on high-frequency brightness change, which can be used to provide data support for the emergency shutdown of the air compressor; the abnormal feature information can also be the foreign matter mixing situation determined by abnormal detection, which can be used to provide data support for the acoustic and optical alarm of the stamping groove and the quality traceability of debris. The spatio-temporal association information can be features such as the regional association degree of debris determined by graph neural network modeling, which can be used to optimize the cooperation strategy of multiple nozzles of the air compressor.

[0034] S21. According to the time stamp of the acquired monitoring image, determine the multi-modal data of the stamping groove corresponding to the time stamp.

[0035] In an embodiment of the present application, each monitoring image in the monitoring video corresponds to a timestamp, which is used to represent the time when the monitoring image is acquired. To improve the efficiency and accuracy of stamping groove cleaning, corresponding multimodal data can also be determined according to the timestamp of the acquired monitoring image. Among them, the multimodal data can be used to provide data support for controlling the air compressor from dimensions other than the image. Exemplarily, the multimodal data can be pneumatic system data, mechanical state data, stamping equipment data, or environmental parameters. Specifically, the pneumatic system data can be the nozzle parameters of the air compressor acquired based on a piezoelectric pressure sensor, the instantaneous value of the air flow pressure of the air compressor acquired based on a thermal mass flowmeter, or the air tank pressure of the air compressor acquired based on an absolute pressure sensor. The mechanical state data can be the cylinder vibration data acquired based on an accelerometer, the solenoid valve response delay data acquired based on a digital monitoring device, or the pipeline temperature gradient data acquired based on an infrared thermal imager. The stamping equipment data can be the stamping cycle data acquired by an electronic device, the die temperature data, the lubricating oil pressure, or the debris weight in the stamping groove. The environmental parameters can be the environmental humidity acquired by a capacitive hygrometer, the environmental temperature acquired by a thermometer, or the workshop air pressure data acquired by an atmospheric pressure sensor.

[0036] S22, perform state estimation on the stamping groove according to the monitoring image and the corresponding multimodal data to obtain the debris state in the stamping groove.

[0037] In an embodiment of the present application, when using an air compressor to clean the debris in the stamping groove, to improve the cleaning efficiency of the stamping groove, state estimation can be performed on the stamping groove according to the monitoring image and the corresponding multimodal data, so as to predict the debris state in the stamping groove. And subsequently, the output parameters of the air compressor can be adjusted in real time according to the predicted debris state in the stamping groove, thereby improving the cleaning efficiency.

[0038] In an embodiment of the present application, the state estimation of the stamping groove based on the monitoring image and the corresponding multimodal data to obtain the debris state in the stamping groove includes: processing the monitoring image according to a pre-trained target recognition model to obtain the attribute data of the debris in the monitoring image; determining the cumulative trend corresponding to the debris in the monitoring image at the time stamp according to the attribute data of the debris in multiple monitoring images; predicting the debris state in the stamping groove according to the cumulative trend and the multimodal data. Specifically, the method for training the target recognition model includes: obtaining historical images in a historical monitoring video, where the historical images correspond to label attribute data; inputting the historical images into a pre-constructed initial recognition model to obtain the predicted attribute data corresponding to the historical images; determining a first loss value of the initial recognition model according to the predicted attribute data and the label attribute data; updating the initial recognition model according to the backpropagation algorithm until the first loss value meets a preset condition, and stopping updating the initial recognition model to obtain a target recognition model trained to a converged state. Among them, the preset condition may be that the first loss value is less than or equal to a preset first termination threshold. When the first loss value is less than or equal to the preset termination threshold, it indicates that the difference between the predicted attribute data output by the initial recognition model and the label attribute data is small, so the accuracy of the result output by the initial recognition model is high, and then the update of the initial recognition model can be stopped to obtain a target recognition model trained to a converged state.

[0039] In an embodiment of the present application, in order to improve the accuracy of the target recognition model, the first loss value of the initial recognition model can be determined according to the image information in the historical image and the preset label attribute data. Specifically, determining the first loss value of the initial recognition model according to the predicted attribute data and the label attribute data includes: constructing a first loss function according to the predicted attribute data and the label attribute data, and determining the first loss value according to the first loss function, and the first loss function satisfies the following relational expression:

[0040] ;

[0041] Among them, Loss1 represents the first loss value of the initial recognition model; i is the index of the historical image, and m is the number of historical images; j is the index of the debris in the historical image indicated by the label attribute data, and n is the number of debris in the historical image indicated by the label attribute data; is the abscissa of the center point of the debris with index j in the i-th historical image indicated by the label attribute data, is the abscissa of the center point of the debris with index j in the i-th historical image indicated by the predicted attribute data; is the ordinate of the center point of the debris with index j in the i-th historical image indicated by the label attribute data, is the ordinate of the center point of the debris with index j in the i-th historical image indicated by the predicted attribute data; is the number of pixel points occupied by the debris with index j in the i-th historical image indicated by the label attribute data, is the number of pixel points occupied by the debris with index j in the i-th historical image indicated by the predicted attribute data; is the probability value that the debris with index j in the i-th historical image belongs to the k-th category indicated by the label attribute data, is the probability value that the debris with index j in the i-th historical image belongs to the k-th category indicated by the predicted attribute data; k is the index of the category corresponding to the debris, and f is the number of categories corresponding to the debris.

[0042] In an embodiment of the present application, determining the cumulative trend corresponding to the debris in the monitoring image at the time stamp according to the attribute data of the debris in multiple monitoring images includes: determining the moving distance and moving direction corresponding to the debris in the stamping groove at each time stamp according to the attribute data of the debris in any two adjacent monitoring images; determining the cumulative trend corresponding to the debris in the monitoring image at the time stamp according to the multi-modal data corresponding to each time stamp and the moving distance and moving direction corresponding to the debris in the stamping groove at each time stamp; the cumulative trend is used to indicate the cumulative weight and cumulative range corresponding to each type of debris.

[0043] Among them, when determining the moving distance and moving direction corresponding to the debris at each time stamp according to the attribute data of the debris in any two adjacent monitoring images, first, image registration can be performed according to the internal parameters of the imaging device and the preset distortion coefficient. Specifically, key points can be extracted from the two frames of images first, and then the matching feature point pairs can be screened according to the Hamming distance, and the affine transformation matrix between the two frames of images can be solved according to the feature point pairs to eliminate the displacement caused by camera jitter or mechanical vibration. The registered images are processed by using an optical flow algorithm or a pre-trained deep learning model to obtain a pixel-level motion vector field, and data such as the moving direction, moving distance, moving speed, and moving acceleration of the debris in the image are determined according to the motion vector field.

[0044] In an embodiment of the present application, predicting the state of the debris in the stamping groove according to the cumulative trend and the multi-modal data includes: determining the state data corresponding to the time stamp according to the cumulative trend and the multi-modal data; predicting the state of the debris in the stamping groove corresponding to the next time stamp according to the state data corresponding to the time stamp and the output parameters of the air compressor corresponding to the time stamp, including:

[0045] ; wherein, represents the debris state in the stamping groove corresponding to the time stamp t + 1; represents the state data in the stamping groove corresponding to the time stamp t; represents the output parameter of the air compressor at time t, represents the weight matrix of the output parameter of the air compressor at time stamp t; represents the error data at time stamp t set in advance.

[0046] S23. Determine the output parameter of the air compressor according to the debris state.

[0047] In an embodiment of the present application, the change range of the output parameter of the air compressor can be determined according to the predicted debris state in the stamping groove. Specifically, determining the output parameter of the air compressor according to the debris state includes:

[0048] ;

[0049] wherein, represents the output parameter of the air compressor at time t + 1; represents the output parameter of the air compressor at time t; represents the debris state in the stamping groove at time t - 1; represents the debris state in the stamping groove at time t; e represents the natural constant. Wherein, represents the change range of the debris state in the stamping groove from time t - 1 to time t. The change range of the debris state is normalized using the natural constant, so as to represent the change range of the debris state based on a floating point number between 0 and 1, and the output parameter of the air compressor at time t is adjusted using this change range to obtain the output parameter of the air compressor at time t + 1. Exemplarily, the output parameters of the air compressor include the instantaneous pressure of the compressed air at the nozzle outlet, the duration of a single blow, the interval time of periodic blows, the deflection angle of the blowing direction relative to the debris accumulation surface, the lateral movement rate of the nozzle along the guide rail, etc.

[0050] S24. Control the output state of the air compressor to the stamping groove based on the output parameter.

[0051] In an embodiment of the present application, the output of the air compressor can be controlled based on the output parameter to adjust the output state of the air compressor to the stamping groove.

[0052] As can be seen from the above technical solutions, in the embodiment of the present application, the attribute data of the debris is first determined based on multiple monitoring images in the monitoring video, so as to improve the detection efficiency of the debris in the stamping groove. Then, according to the attribute data of the debris and the multi-modal data obtained in real time, the state of the debris in the stamping groove is determined. In this way, the position change and the change of the accumulation trend of the debris in the stamping groove can be characterized by the debris state, and the correlation relationship between the output parameters of the air compressor and the debris state in the time dimension is determined according to the debris states corresponding to different timestamps. In this way, the influence degree of the output parameters of the air compressor on the debris state in the stamping groove can be determined in the time dimension and the space dimension, and finally the output parameters of the air compressor are updated according to the predicted debris state. In this way, the output state of the air compressor can be adjusted based on the data related to the debris in the stamping groove obtained in real time, and thus the cleaning efficiency and accuracy of the stamping groove can be improved.

[0053] Please refer to Figure 3 , Figure 3 which is a functional module diagram of a stamping groove cleaning system based on machine vision provided by an embodiment of the present application. The stamping groove cleaning system 500 based on machine vision includes an electronic device 100, and the electronic device 100 is communicatively connected to an air compressor 200. The module / unit referred to in the present application means a series of computer-readable instruction segments that can be executed by a processor 13 and can complete a fixed function, and is stored in a memory 12. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0054] The electronic device 100 is configured to obtain a monitoring video of the stamping groove in real time, and the monitoring video includes at least one monitoring image.

[0055] The electronic device 100 is further configured to determine the multi-modal data of the stamping groove corresponding to the timestamp according to the timestamp for obtaining the monitoring image.

[0056] The electronic device 100 is further configured to perform state estimation on the stamping groove according to the monitoring image and the corresponding multi-modal data to obtain the debris state in the stamping groove.

[0057] The electronic device 100 is further configured to determine the output parameters of the air compressor 200 according to the debris state.

[0058] The electronic device 100 is further configured to control the output state of the air compressor 200 on the stamping groove based on the output parameters.

[0059] In some embodiments, the electronic device 100 is further configured to process the monitored image according to a pre-trained target recognition model to obtain attribute data of the debris in the monitored image; determine the cumulative trend of the debris in the monitored image corresponding to the time stamp according to the attribute data of the debris in multiple monitored images; and predict the debris state in the stamping groove according to the cumulative trend and the multimodal data.

[0060] In some embodiments, the electronic device 100 is further configured to obtain historical images in a historical monitored video, where the historical images correspond to label attribute data; input the historical images into a pre-constructed initial recognition model to obtain predicted attribute data corresponding to the historical images; determine a first loss value of the initial recognition model according to the predicted attribute data and the label attribute data; and update the initial recognition model according to the backpropagation algorithm until the first loss value meets a preset condition, and then stop updating the initial recognition model to obtain a target recognition model trained to a converged state.

[0061] In some embodiments, the electronic device 100 is further configured to construct a first loss function according to the predicted attribute data and the label attribute data, and determine the first loss value according to the first loss function. The first loss function satisfies the following relational expression:

[0062] ; where Loss1 represents the first loss value of the initial recognition model; i is the index of the historical image, m is the number of historical images; j is the index of the debris in the historical image indicated by the label attribute data, and n is the number of debris in the historical image indicated by the label attribute data; is the abscissa of the center point of the debris with index j in the i-th historical image indicated by the label attribute data, is the abscissa of the center point of the debris with index j in the i-th historical image indicated by the predicted attribute data; is the ordinate of the center point of the debris with index j in the i-th historical image indicated by the label attribute data, is the ordinate of the center point of the debris with index j in the i-th historical image indicated by the predicted attribute data; is the number of pixel points occupied by the debris with index j in the i-th historical image indicated by the label attribute data, is the number of pixel points occupied by the debris with index j in the i-th historical image indicated by the predicted attribute data; is the probability value that the debris with index j in the i-th historical image indicated by the label attribute data belongs to the k-th category, is the probability value that the debris with index j in the i-th historical image indicated by the predicted attribute data belongs to the k-th category; where k is the index of the category corresponding to the debris, and f is the number of categories corresponding to the debris.

[0063] In some embodiments, the electronic device 100 is further configured to determine the moving distance and moving direction corresponding to each timestamp of the debris in the stamping groove according to the attribute data of the debris in any two adjacent monitoring images; determine the cumulative trend corresponding to the debris in the monitoring image at each timestamp according to the multi-modal data corresponding to each timestamp and the moving distance and moving direction corresponding to the debris in the stamping groove at each timestamp; the cumulative trend is used to indicate the cumulative weight and cumulative range corresponding to each type of debris.

[0064] In some embodiments, the electronic device 100 is further configured to determine the state data corresponding to the timestamp according to the cumulative trend and the multi-modal data; predict the state of the debris in the stamping groove corresponding to the next timestamp according to the state data corresponding to the timestamp and the output parameters of the air compressor corresponding to the timestamp, including:

[0065] ; where represents the state of the debris in the stamping groove corresponding to the timestamp t + 1; represents the state data of the stamping groove corresponding to the timestamp t; represents the output parameter of the air compressor at time t, represents the weight matrix of the output parameter of the air compressor at time t; represents the error data at time t preset.

[0066] In some embodiments, the electronic device 100 determines the output parameter of the air compressor according to the debris state, including:

[0067] ; where represents the output parameter of the air compressor at time t + 1; represents the output parameter of the air compressor at time t; represents the state of the debris in the stamping groove at time t - 1; represents the state of the debris in the stamping groove at time t; e represents the natural constant.

[0068] Please refer to Figure 4, which is a schematic structural diagram of an electronic device provided by an embodiment of the present application. The electronic device 100 includes a memory 12 and a processor 13. The memory 12 is used to store computer-readable instructions, and the processor 13 is used to execute the computer-readable instructions stored in the memory to implement a stamping groove cleaning method based on machine vision described in any of the above embodiments.

[0069] In an embodiment of the present application, the electronic device 100 further includes a bus and a computer program stored in the memory 12 and executable on the processor 13, such as a stamping groove cleaning program based on machine vision.

[0070] Figure 4 Only the electronic device 100 with the memory 12 and the processor 13 is shown. Those skilled in the art can understand that Figure 4 the shown structure does not limit the electronic device 100, and it may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0071] Combined with Figure 2 , the memory 12 in the electronic device 100 stores multiple computer-readable instructions to implement the stamping groove cleaning method based on machine vision. The processor 13 can execute the multiple instructions to achieve: real-time acquisition of a monitoring video of the stamping groove, where the monitoring video includes at least one monitoring image; determining multi-modal data of the stamping groove corresponding to the time stamp according to the time stamp of obtaining the monitoring image; the multi-modal data at least includes the debris weight in the stamping groove and the air flow pressure of the air compressor; performing state estimation on the stamping groove according to the monitoring image and the corresponding multi-modal data to obtain the debris state in the stamping groove; determining the output parameters of the air compressor according to the debris state; and controlling the output state of the air compressor to the stamping groove based on the output parameters.

[0072] Specifically, the specific implementation method of the above instructions by the processor 13 can refer to Figure 3 the description of the relevant steps in the corresponding embodiment, which will not be elaborated here.

[0073] Those skilled in the art can understand that the schematic diagram is only an example of the electronic device 100 and does not limit the electronic device 100. The electronic device 100 can be a bus structure or a star structure. The electronic device 100 can also include more or fewer other hardware or software than shown, or different component arrangements. For example, the electronic device 100 can also include input / output devices, network access devices, etc.

[0074] It should be noted that the electronic device 100 is only an example. Other existing or future electronic products that can be adapted to this application should also be included within the protection scope of this application and are hereby incorporated by reference.

[0075] Among them, the memory 12 includes at least one type of readable storage medium. The readable storage medium can be non-volatile or volatile. The readable storage medium includes flash memory, mobile hard disks, multimedia cards, card-type memories (such as SD or DX memories, etc.), magnetic memories, magnetic disks, optical disks, etc. The memory 12 can be an internal storage unit of the electronic device 100 in some embodiments, such as the mobile hard disk of the electronic device 100. The memory 12 can also be an external storage device of the electronic device 100 in other embodiments, such as a plug-in mobile hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a FlashCard, etc. equipped on the electronic device 100. The memory 12 can be used not only to store application software installed on the electronic device 100 and various types of data, such as the code of a stamping groove cleaning program based on machine vision, etc., but also to temporarily store data that has been output or will be output.

[0076] The processor 13 can be composed of integrated circuits in some embodiments. For example, it can be composed of a single packaged integrated circuit, or can be composed of multiple integrated circuits with the same or different functions, including the combination of one or more Central Processing Units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips, etc. The processor 13 is the control core (Control Unit) of the electronic device 100, connecting various components of the entire electronic device 100 through various interfaces and circuits. By running or executing programs or modules stored in the memory 12 (such as executing a stamping groove cleaning program based on machine vision), and calling data stored in the memory 12, it executes various functions of the electronic device 100 and processes data.

[0077] The processor 13 executes the operating system of the electronic device 100 and various installed application programs. The processor 13 executes the application programs to implement the steps in the above-described embodiments of various stamping groove cleaning methods based on machine vision, such as Figure 2 the steps shown.

[0078] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete the present application. The one or more modules / units may be a series of computer-readable instruction segments capable of performing specific functions, and these instruction segments are used to describe the execution process of the computer program in the electronic device 100.

[0079] The integrated units implemented in the form of software function modules as described above may be stored in a computer-readable storage medium. The above-mentioned software function modules stored in a storage medium include several instructions for causing a computer device (which may be a personal computer, a computer device, or a network device, etc.) or a processor (Processor) to execute a part of the method for cleaning a stamping groove based on machine vision described in various embodiments of the present application.

[0080] If the modules / units integrated in the electronic device 100 are implemented in the form of software function units and sold or used as independent products, they may be stored in a computer-readable storage medium. Based on such an understanding, to implement all or part of the processes in the above-mentioned method embodiments of the present application, it may also be completed by a computer program instructing relevant hardware devices. The computer program may be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above-mentioned method embodiments may be implemented.

[0081] Among them, the computer program includes computer program code, and the computer program code may be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disc, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory, and other memories, etc.

[0082] Furthermore, the computer-readable storage medium mainly includes a program storage area and a data storage area. Among them, the program storage area may store an operating system, application programs required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0083] The bus may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus may be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, in Figure 4It is represented by only one arrow in the figure, but it does not mean that there is only one bus or one type of bus. The bus is set to realize the connection and communication between the memory 12 and at least one processor 13, etc.

[0084] An embodiment of the present application also provides a computer-readable storage medium (not shown in the figure). Computer-readable instructions are stored in the computer-readable storage medium, and the computer-readable instructions are executed by a processor in an electronic device to implement the method for cleaning a stamping groove based on machine vision described in any of the above embodiments.

[0085] In several embodiments provided by the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0086] The modules described as separate components may or may not be physically separated. The components shown as modules may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0087] In addition, in each embodiment of the present application, the functional modules can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware, or in the form of a combination of hardware and software functional modules.

[0088] In addition, obviously, the word "including" does not exclude other units or steps, and the singular does not exclude the plural. The multiple units or devices described in the specification can also be implemented by one unit or device through software or hardware. Words such as first and second are used to represent names and do not represent any specific order.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit them. Although the present application has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present application can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. A stamping slot cleaning method based on machine vision, applied to an electronic device, wherein the electronic device is communicatively connected to an air compressor, characterized in that: The method comprises: Acquire a monitoring video of the punching slot in real time, wherein the monitoring video includes at least one monitoring image; Determining, according to a timestamp of acquiring the monitoring image, multimodal data of the punching slot corresponding to the timestamp; The state of the stamping slot is estimated according to the monitoring image and the corresponding multimodal data to obtain the state of debris in the stamping slot, including: processing the monitoring image according to a pre-trained target recognition model to obtain attribute data of the debris in the monitoring image; determining the cumulative trend of the debris in the monitoring image corresponding to the timestamp according to the attribute data of the debris in multiple monitoring images; predicting the state of the debris in the stamping slot according to the cumulative trend and the multimodal data; According to the state of the debris, the output parameters of the air compressor are determined, including: ;in, Represents the output parameters of the space compressor at timestamp t+1; Represents the output parameters of the air compressor at timestamp t; represents the state of the debris in the punch slot at the time stamp t-1; represents the state of debris in the punch slot at time stamp t; e represents a natural constant; Based on the output parameter, the output state of the air compressor to the punching groove is controlled.

2. The method for cleaning a stamping groove based on machine vision according to claim 1, characterized in that: The method further includes training the target recognition model, wherein the training the target recognition model includes: Acquire historical images in historical surveillance videos, wherein the historical images correspond to tag attribute data; Inputting the historical image into a pre-built initial recognition model to obtain predicted attribute data corresponding to the historical image; Determining a first loss value of the initial recognition model according to the predicted attribute data and the label attribute data; The initial recognition model is updated according to the back propagation algorithm until the first loss value meets a preset condition, and the updating of the initial recognition model is stopped to obtain a target recognition model trained to a convergent state.

3. The machine vision-based stamping groove cleaning method according to claim 2, characterized in that: Determining the first loss value of the initial recognition model according to the predicted attribute data and the label attribute data includes: A first loss function is constructed according to the predicted attribute data and the label attribute data, and the first loss value is determined according to the first loss function. The first loss function satisfies the following relationship: ； Wherein, Loss1 represents the first loss value of the initial recognition model; i is the index of the historical image, m is the number of the historical images; j is the index of the debris in the historical image indicated by the label attribute data, and n is the number of debris in the historical image indicated by the label attribute data; is the abscissa of the center point of the debris with index j in the ith historical image indicated by the label attribute data, is the abscissa of the center point of the debris with index j in the ith historical image indicated by the prediction attribute data; is the ordinate of the center point of the debris with index j in the i-th historical image indicated by the label attribute data, is the ordinate of the center point of the debris with index j in the i-th historical image indicated by the predicted attribute data; is the number of pixels occupied by debris with index j in the i-th historical image indicated by the label attribute data, is the number of pixels occupied by debris with index j in the i-th historical image indicated by the predicted attribute data; is the probability value that the debris with index j in the ith historical image indicated by the label attribute data belongs to the kth category, is the probability value that the debris with index j in the ith historical image indicated by the predicted attribute data belongs to the kth category; k is the index of the category corresponding to the debris, and f is the number of categories corresponding to the debris.

4. The method for cleaning a stamping groove based on machine vision according to claim 1, characterized in that: Determining the accumulation trend of the debris in the monitoring images corresponding to the timestamp according to the attribute data of the debris in the multiple monitoring images includes: Determine the moving distance and moving direction of the debris in the punching slot corresponding to each time stamp according to the attribute data of the debris in any two adjacent monitoring images; According to the multimodal data corresponding to each timestamp and the moving distance and moving direction of the debris in the stamping groove corresponding to each timestamp, the cumulative trend of the debris in the monitoring image corresponding to the timestamp is determined; the cumulative trend is used to indicate the cumulative weight and cumulative range corresponding to each type of debris.

5. The method for cleaning a stamping groove based on machine vision according to claim 1, characterized in that: The predicting of the chip state in the punching slot according to the cumulative trend and the multimodal data comprises: Determining state data corresponding to the timestamp according to the cumulative trend and the multimodal data; Predicting the state of the chips in the punching groove corresponding to the next time stamp according to the state data corresponding to the time stamp and the output parameter of the air compressor corresponding to the time stamp includes: ； in, represents the state of the debris in the punching groove corresponding to the time stamp t+1; Represents the state data in the punching slot corresponding to the time stamp t; represents the output parameter of the air compressor at time t, A weight matrix representing the output parameters of the air compressor at timestamp t; Represents the error data at the preset timestamp t.

6. A punching slot cleaning system based on machine vision, characterized in that: The system includes an electronic device, the electronic device being communicatively connected to the air compressor; The electronic device is used to obtain a monitoring video of the punching slot in real time, wherein the monitoring video includes at least one monitoring image; The electronic device is also used to determine the multimodal data of the punching slot corresponding to the timestamp according to the timestamp of acquiring the monitoring image; The electronic device is also used to estimate the state of the stamping slot according to the monitoring image and the corresponding multimodal data to obtain the state of debris in the stamping slot, including: processing the monitoring image according to a pre-trained target recognition model to obtain attribute data of the debris in the monitoring image; determining the cumulative trend of the debris in the monitoring image corresponding to the timestamp according to the attribute data of the debris in multiple monitoring images; predicting the state of the debris in the stamping slot according to the cumulative trend and the multimodal data; The electronic device is also used to determine the output parameters of the air compressor according to the state of the debris, including: ;in, Represents the output parameters of the space compressor at timestamp t+1; Represents the output parameters of the air compressor at timestamp t; represents the state of the debris in the punch slot at the time stamp t-1; represents the state of debris in the punch slot at time stamp t; e represents a natural constant; The electronic device is also used to control the output state of the air compressor to the punching groove based on the output parameter.

7. The machine vision-based punching slot cleaning system according to claim 6, characterized in that: The electronic device is also used for: Acquire historical images in historical surveillance videos, wherein the historical images correspond to tag attribute data; Inputting the historical image into a pre-built initial recognition model to obtain predicted attribute data corresponding to the historical image; Determining a first loss value of the initial recognition model according to the predicted attribute data and the label attribute data; The initial recognition model is updated according to the back propagation algorithm until the first loss value meets a preset condition, and the updating of the initial recognition model is stopped to obtain a target recognition model trained to a convergent state.

Images