Stamping slot cleaning method and system based on machine vision

Abstract

The present application proposes a stamping slot cleaning method and system based on machine vision. The stamping slot cleaning method based on machine vision includes: acquiring a monitoring video of the stamping slot in real time, wherein the monitoring video includes at least one monitoring image; determining the multimodal data of the stamping slot corresponding to the timestamp according to the timestamp of acquiring the monitoring image; the multimodal data includes at least the weight of the debris in the stamping slot and the airflow pressure of the air compressor; estimating the state of the stamping slot according to the monitoring image and the corresponding multimodal data to obtain the state of the debris in the stamping slot; determining the output parameters of the air compressor according to the state of the debris; and controlling the output state of the air compressor to the stamping slot based on the output parameters. The present application relates to the field of intelligent manufacturing technology and can improve the efficiency of stamping slot cleaning.

Description

Technical Field

[0001] The present application relates to the field of intelligent manufacturing technology, and in particular to a stamping groove cleaning method and system based on machine vision. Background Art

[0002] At present, with the development of data science, some enterprises or organizations tend to improve productivity and the efficiency of production equipment maintenance through data analysis in the production environment. In the production and maintenance process of the stamping production line, air compressors are usually used to clean the debris in the stamping slot. In order to improve the cleaning efficiency, cleaning strategies are usually formulated based on digital control technology and image recognition technology. However, this method is not adaptable enough to dynamic environments, and it is impossible to adjust the debris detection strategy according to light changes and the diversity of debris morphology, which can easily lead to problems of miscleaning or missed detection. Moreover, when using a single modal data to assist cleaning, the timing characteristics of the debris accumulation process and the environmental information in the production environment are not utilized, which may lead to excessive blowing (energy waste) or incomplete cleaning, resulting in low cleaning efficiency. Summary of the invention

[0003] In view of the above, it is necessary to propose a stamping groove cleaning method and system based on machine vision to solve the technical problems of low efficiency and low accuracy in cleaning the stamping groove.

[0004] The present application provides a stamping slot cleaning method based on machine vision, which is applied to an electronic device, wherein the electronic device is communicatively connected to an air compressor, and the method comprises: acquiring a monitoring video of the stamping slot in real time, wherein the monitoring video comprises at least one monitoring image; determining multimodal data of the stamping slot corresponding to the timestamp of acquiring the monitoring image; estimating the state of the stamping slot according to the monitoring image and the corresponding multimodal data to obtain a debris state in the stamping slot; determining an output parameter of the air compressor according to the debris state; and controlling the output state of the air compressor to the stamping slot based on the output parameter.

[0005] In some embodiments, the state of the stamping slot is estimated based on the monitoring image and the corresponding multimodal data to obtain the state of debris in the stamping slot, including: processing the monitoring image according to a pre-trained target recognition model to obtain attribute data of the debris in the monitoring image; determining the cumulative trend of the debris in the monitoring image corresponding to the timestamp based on the attribute data of the debris in multiple monitoring images; and predicting the state of debris in the stamping slot based on the cumulative trend and the multimodal data.

[0006] In some embodiments, the method also includes training the target recognition model, and the training of the target recognition model includes: obtaining historical images in historical surveillance videos, wherein the historical images correspond to label attribute data; inputting the historical images into a pre-constructed initial recognition model to obtain predicted attribute data corresponding to the historical images; determining a first loss value of the initial recognition model based on the predicted attribute data and the label attribute data; updating the initial recognition model according to a back-propagation algorithm until the first loss value meets a preset condition, stopping updating the initial recognition model, and obtaining a target recognition model trained to a convergence state.

[0007] In some embodiments, determining the first loss value of the initial recognition model according to the predicted attribute data and the label attribute data includes: constructing a first loss function according to the predicted attribute data and the label attribute data, and determining the first loss value according to the first loss function, wherein the first loss function satisfies the following relationship:

[0008] ; Wherein, Loss1 represents the first loss value of the initial recognition model; i is the index of the historical image, m is the number of the historical images; j is the index of the debris in the historical image indicated by the label attribute data, and n is the number of debris in the historical image indicated by the label attribute data; is the horizontal coordinate of the center point of the debris with index j in the i-th historical image indicated by the label attribute data, is the abscissa of the center point of the debris with index j in the i-th historical image indicated by the predicted attribute data; is the ordinate of the center point of the debris with index j in the i-th historical image indicated by the label attribute data, is the ordinate of the center point of the debris with index j in the i-th historical image indicated by the predicted attribute data; is the number of pixels occupied by debris with index j in the i-th historical image indicated by the label attribute data, is the number of pixels occupied by debris with index j in the i-th historical image indicated by the predicted attribute data; is the probability value that the debris with index j in the ith historical image indicated by the label attribute data belongs to the kth category, is the probability value that the debris with index j in the ith historical image indicated by the predicted attribute data belongs to the kth category; k is the index of the category corresponding to the debris, and f is the number of categories corresponding to the debris.

[0009] In some embodiments, determining the cumulative trend of the debris in the monitoring images corresponding to the timestamp based on the attribute data of the debris in multiple monitoring images includes: determining the movement distance and movement direction of the debris in the stamping groove corresponding to each timestamp based on the attribute data of the debris in any two adjacent monitoring images; determining the cumulative trend of the debris in the monitoring images corresponding to the timestamp based on the multimodal data corresponding to each timestamp and the movement distance and movement direction of the debris in the stamping groove corresponding to each timestamp; the cumulative trend is used to indicate the cumulative weight and cumulative range corresponding to each type of debris.

[0010] In some embodiments, predicting the state of debris in the punching slot according to the cumulative trend and the multimodal data includes: determining the state data corresponding to the timestamp according to the cumulative trend and the multimodal data; predicting the state of debris in the punching slot corresponding to the next timestamp according to the state data corresponding to the timestamp and the output parameter of the air compressor corresponding to the timestamp, including:

[0011] ;in, represents the state of the debris in the punching groove corresponding to the time stamp t+1; Represents the state data in the punching slot corresponding to the time stamp t; represents the output parameter of the air compressor at time t, A weight matrix representing the output parameters of the air compressor at timestamp t; Represents the error data at the preset timestamp t.

[0012] In some embodiments, determining the output parameter of the air compressor according to the debris state includes:

[0013] ;in, Represents the output parameters of the space compressor at timestamp t+1; Represents the output parameters of the air compressor at timestamp t; represents the state of the debris in the punch slot at the time stamp t-1; represents the state of debris in the punch groove at timestamp t; e represents a natural constant.

[0014] An embodiment of the present application also provides a machine vision-based stamping slot cleaning system, the system comprising an electronic device, the electronic device being communicatively connected to an air compressor; the electronic device being used to acquire a monitoring video of the stamping slot in real time, the monitoring video comprising at least one monitoring image; the electronic device being further used to determine, based on a timestamp at which the monitoring image is acquired, multimodal data of the stamping slot corresponding to the timestamp; the electronic device being further used to perform state estimation on the stamping slot based on the monitoring image and the corresponding multimodal data to obtain a debris state in the stamping slot; the electronic device being further used to determine an output parameter of the air compressor based on the debris state; the electronic device being further used to control the output state of the air compressor to the stamping slot based on the output parameter.

[0015] In some embodiments, the electronic device is also used to: process the monitoring image according to a pre-trained target recognition model to obtain attribute data of the debris in the monitoring image; determine the cumulative trend of the debris in the monitoring image corresponding to the timestamp based on the attribute data of the debris in multiple monitoring images; and predict the state of the debris in the stamping groove based on the cumulative trend and the multimodal data.

[0016] In some embodiments, the electronic device is also used to: obtain historical images in historical surveillance videos, wherein the historical images correspond to label attribute data; input the historical images into a pre-built initial recognition model to obtain predicted attribute data corresponding to the historical images; determine a first loss value of the initial recognition model based on the predicted attribute data and the label attribute data; update the initial recognition model according to a back-propagation algorithm until the first loss value meets a preset condition, stop updating the initial recognition model, and obtain a target recognition model trained to a convergence state.

[0017] It can be seen from the above technical solution that the embodiment of the present application first determines the attribute data of the debris based on multiple monitoring images in the monitoring video, so as to improve the efficiency of detecting debris in the stamping slot. Then, the state of the debris in the stamping slot is determined based on the attribute data of the debris and the multimodal data acquired in real time. In this way, the debris state can be used to characterize the position change and cumulative trend change of the debris in the stamping slot, and the correlation between the output parameters of the air compressor and the debris state in the time dimension can be determined according to the debris state corresponding to different timestamps. In this way, the degree of influence of the output parameters of the air compressor on the debris state in the stamping slot can be determined in the time dimension and the space dimension, and finally the output parameters of the air compressor are updated according to the predicted debris state. In this way, the output state of the air compressor can be adjusted based on the data related to the debris in the stamping slot acquired in real time, thereby improving the cleaning efficiency and accuracy of the stamping slot. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1This is an application scenario diagram of a stamping groove cleaning method based on machine vision provided in one embodiment of the present application.

[0019] Figure 2 This is a flow chart of a stamping groove cleaning method based on machine vision provided in one embodiment of the present application.

[0020] Figure 3 This is a functional module diagram of a stamping groove cleaning device based on machine vision provided in one embodiment of the present application.

[0021] Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to more clearly understand the purpose, features and advantages of the present application, the present application is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other. In the following description, many specific details are set forth to facilitate a full understanding of the present application, and the embodiments described are only a part of the embodiments of the present application, rather than all of the embodiments.

[0023] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by those skilled in the art to which the present application belongs. The terms used herein in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0025] An embodiment of the present application provides a stamping groove cleaning method based on machine vision, which can be applied to one or more electronic devices. The electronic device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0026] An electronic device can be any electronic product that can interact with customers, such as personal computers, tablet computers, smart phones, personal digital assistants (PDAs), game consoles, Internet Protocol Television (IPTV), smart wearable devices, etc.

[0027] The electronic device may also include a network device and / or a client device, wherein the network device includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud consisting of a large number of hosts or network servers based on cloud computing.

[0028] The networks where electronic devices are located include but are not limited to the Internet, wide area network, metropolitan area network, local area network, virtual private network (VPN), etc.

[0029] like Figure 1 The figure shows an application scenario diagram of a stamping slot cleaning method based on machine vision provided by an embodiment of the present application. The stamping slot cleaning method based on machine vision provided by the present application can be applied to an electronic device 100. Among them, the electronic device 100 is communicatively connected to the air compressor 200 and the camera device 300. The electronic device 100 acquires the monitoring video of the stamping slot 400 taken by the camera device 300 in real time, and the monitoring video includes at least one monitoring image. The electronic device 100 also determines the multimodal data of the stamping slot 400 corresponding to the timestamp of acquiring the monitoring image, and estimates the state of the stamping slot 400 according to the monitoring image and the corresponding multimodal data, and obtains the debris state in the stamping slot 400. The electronic device 100 also determines the output parameters of the air compressor 200 according to the debris state, and controls the output state of the air compressor 200 to the stamping slot 400 based on the output parameters.

[0030] like Figure 2 , is a flow chart of a method for cleaning a stamping groove based on machine vision provided by an embodiment of the present application. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted. A method for cleaning a stamping groove based on machine vision provided by an embodiment of the present application includes the following steps.

[0031] S20, acquiring a monitoring video of the punching slot in real time, wherein the monitoring video includes at least one monitoring image.

[0032] In one embodiment of the present application, when cleaning a stamping slot in an industrial production environment, a monitoring video of the stamping slot can be obtained in real time, and the monitoring video includes at least one monitoring image. The information of debris in the stamping slot is subsequently extracted through the video monitoring image. The debris information in the stamping slot includes multiple dimensions, which can provide a data basis for the decision of debris removal.

[0033] Exemplarily, the debris information in the stamping slot includes: geometric feature information, dynamic behavior information, physical quantity information, abnormal feature information, and spatiotemporal correlation information. Specifically, the geometric feature information can be a debris distribution heat map based on kernel density analysis of density estimation, which can be used to optimize the nozzle layout and purge path planning of the air compressor; the geometric feature information can also be the accumulation morphology data of the debris obtained based on three-dimensional point cloud reconstruction, which can be used to predict the collapse risk of the debris in the stamping slot and predict the debris removal priority; the geometric feature information can also be the coverage area of ​​the debris based on pixel statistics obtained after semantic segmentation, which can be used to trigger a graded alarm threshold; the geometric feature information can also be the projection contour of the debris determined based on the edge detection algorithm, which can be used to judge the shape characteristics of the debris (for example, needle-shaped, flake-shaped or granular). For example, long strips of metal debris can be identified through contour analysis, so that the blowing angle can be automatically adjusted to prevent the debris from being entangled. Dynamic behavior information can be the movement trajectory of debris obtained based on the optical flow method, which can be used to predict the source of debris generation; dynamic behavior information can also be the debris splashing speed obtained based on the target tracking algorithm, which can be used to calculate the impact risk of the kinetic energy of the debris on the stamping slot; dynamic behavior information can also be the accumulation rate of debris obtained by time-series difference image analysis, which can be used to predict the maintenance cycle of the stamping slot. For example, when the horizontal speed of the debris exceeds 3.2m / s, a predictive purge strategy can be adopted to improve the removal efficiency. Physical quantity information can be the equivalent weight of the debris determined based on the volume density integral method; physical quantity information can also be the accumulation angle of the debris calculated based on the minimum bounding box inclination angle; physical quantity information can also be the particle size distribution of the debris determined based on the watershed algorithm; physical quantity information can also be the surface roughness of the debris obtained by texture analysis based on the grayscale co-occurrence matrix. Abnormal feature information can be the spark splashing abnormality of debris obtained based on high-frequency brightness change detection, which can be used to provide data support for the emergency shutdown of the air compressor; abnormal feature information can also be the mixing of foreign matter determined based on abnormal detection, which can be used to provide data support for the sound and light alarm of the stamping slot and the quality traceability of the debris. Spatiotemporal correlation information can be the regional correlation degree of debris determined based on graph neural network modeling, which can be used to optimize the coordination strategy of multiple nozzles of the air compressor.

[0034] S21, determining, according to the timestamp of acquiring the monitoring image, the multimodal data of the stamping groove corresponding to the timestamp.

[0035] In one embodiment of the present application, each monitoring image in the monitoring video corresponds to a timestamp, which is used to characterize the time when the monitoring image was acquired. In order to improve the efficiency and accuracy of the stamping slot cleaning, the corresponding multimodal data can also be determined according to the timestamp of the monitoring image. Among them, the multimodal data can be used to provide data support for controlling the air compressor from dimensions other than the image. Exemplarily, the multimodal data can be air circuit system data, mechanical state data, stamping equipment data, or environmental parameters. Specifically, the air circuit system data can be the nozzle parameters of the air compressor obtained based on the piezoelectric pressure sensor, the air circuit system data can also be the instantaneous value of the air flow pressure of the air compressor obtained based on the thermal mass flowmeter, and the air circuit system data can also be the gas tank pressure of the air compressor obtained based on the absolute pressure sensor. The mechanical state data can be the cylinder vibration data obtained based on the accelerometer, the mechanical state data can also be the solenoid valve response delay data obtained based on the digital monitoring device, and the mechanical state data can also be the pipeline temperature gradient data obtained based on the infrared thermal imager. The stamping equipment data can be stamping cycle data obtained by electronic equipment, mold temperature data, lubricating oil pressure, or chip weight in the stamping groove. The environmental parameter can be the ambient humidity obtained by a capacitive hygrometer, the ambient temperature obtained by a thermometer, or the workshop air pressure data obtained by an atmospheric pressure sensor.

[0036] S22, estimating the state of the stamping slot according to the monitoring image and the corresponding multimodal data to obtain the state of debris in the stamping slot.

[0037] In one embodiment of the present application, when using an air compressor to clean the debris in the punching slot, in order to improve the cleaning efficiency of the punching slot, the state of the punching slot can be estimated based on the monitoring image and the corresponding multimodal data, thereby predicting the state of the debris in the punching slot. The output parameters of the air compressor are subsequently adjusted in real time according to the predicted state of the debris in the punching slot, thereby improving the cleaning efficiency.

[0038] In one embodiment of the present application, the state estimation of the stamping slot according to the monitoring image and the corresponding multimodal data to obtain the state of the debris in the stamping slot includes: processing the monitoring image according to a pre-trained target recognition model to obtain the attribute data of the debris in the monitoring image; determining the cumulative trend of the debris in the monitoring image corresponding to the timestamp according to the attribute data of the debris in multiple monitoring images; predicting the state of the debris in the stamping slot according to the cumulative trend and the multimodal data. Specifically, the method for training the target recognition model includes: obtaining historical images in historical monitoring videos, the historical images corresponding to label attribute data; inputting the historical images into a pre-constructed initial recognition model to obtain the predicted attribute data corresponding to the historical images; determining the first loss value of the initial recognition model according to the predicted attribute data and the label attribute data; updating the initial recognition model according to the back propagation algorithm until the first loss value meets the preset condition, stopping updating the initial recognition model, and obtaining the target recognition model trained to a converged state. The preset condition may be that the first loss value is less than or equal to a preset first termination threshold. When the first loss value is less than or equal to the preset termination threshold, it indicates that the difference between the predicted attribute data and the label attribute data output by the initial recognition model is small, and the accuracy of the result output by the initial recognition model is high. In this case, the updating of the initial recognition model can be stopped to obtain the target recognition model trained to a convergent state.

[0039] In one embodiment of the present application, in order to improve the accuracy of the target recognition model, the first loss value of the initial recognition model can be determined based on the image information in the historical image and the pre-set label attribute data. Specifically, determining the first loss value of the initial recognition model based on the predicted attribute data and the label attribute data includes: constructing a first loss function based on the predicted attribute data and the label attribute data, determining the first loss value based on the first loss function, and the first loss function satisfies the following relationship:

[0040] ;

[0041] Wherein, Loss1 represents the first loss value of the initial recognition model; i is the index of the historical image, m is the number of the historical images; j is the index of the debris in the historical image indicated by the label attribute data, and n is the number of debris in the historical image indicated by the label attribute data; is the horizontal coordinate of the center point of the debris with index j in the i-th historical image indicated by the label attribute data, is the abscissa of the center point of the debris with index j in the i-th historical image indicated by the predicted attribute data; is the ordinate of the center point of the debris with index j in the i-th historical image indicated by the label attribute data, is the ordinate of the center point of the debris with index j in the i-th historical image indicated by the predicted attribute data; is the number of pixels occupied by debris with index j in the i-th historical image indicated by the label attribute data, is the number of pixels occupied by debris with index j in the i-th historical image indicated by the predicted attribute data; is the probability value that the debris with index j in the ith historical image indicated by the label attribute data belongs to the kth category, is the probability value that the debris with index j in the ith historical image indicated by the predicted attribute data belongs to the kth category; k is the index of the category corresponding to the debris, and f is the number of categories corresponding to the debris.

[0042] In one embodiment of the present application, determining the cumulative trend of the debris in the monitoring images corresponding to the timestamp based on the attribute data of the debris in multiple monitoring images includes: determining the movement distance and movement direction of the debris in the stamping groove corresponding to each timestamp based on the attribute data of the debris in any two adjacent monitoring images; determining the cumulative trend of the debris in the monitoring image corresponding to the timestamp based on the multimodal data corresponding to each timestamp and the movement distance and movement direction of the debris in the stamping groove corresponding to each timestamp; the cumulative trend is used to indicate the cumulative weight and cumulative range corresponding to each type of debris.

[0043] Among them, when determining the moving distance and moving direction of the debris at each timestamp based on the attribute data of the debris in any two adjacent monitoring images, the image registration can be first performed based on the internal parameters of the camera device and the preset distortion coefficient. Specifically, key points can be extracted from the two frames of images first, and then the matching feature point pairs can be screened based on the Hamming distance. The affine transformation matrix between the two frames of images is solved based on the feature point pairs to eliminate the displacement caused by camera shake or mechanical vibration. The registered images are processed using an optical flow algorithm or a pre-trained deep learning model to obtain a pixel-level motion vector field, and the moving direction, moving distance, moving speed, and moving acceleration of the debris in the image are determined based on the motion vector field.

[0044] In one embodiment of the present application, predicting the state of debris in the punching slot according to the cumulative trend and the multimodal data includes: determining the state data corresponding to the timestamp according to the cumulative trend and the multimodal data; predicting the state of debris in the punching slot corresponding to the next timestamp according to the state data corresponding to the timestamp and the output parameter of the air compressor corresponding to the timestamp, including:

[0045] ;in, represents the state of the debris in the punching groove corresponding to the time stamp t+1; Represents the state data in the punching slot corresponding to the time stamp t; represents the output parameter of the air compressor at time t, A weight matrix representing the output parameters of the air compressor at timestamp t; Represents the error data at the preset timestamp t.

[0046] S23, determining output parameters of the air compressor according to the debris state.

[0047] In one embodiment of the present application, the variation range of the output parameter of the air compressor can be determined according to the predicted state of the debris in the punching groove. Specifically, according to the state of the debris, determining the output parameter of the air compressor includes:

[0048] ;

[0049] in, Represents the output parameters of the space compressor at timestamp t+1; Represents the output parameters of the air compressor at timestamp t; represents the state of the debris in the punch slot at the time stamp t-1; represents the state of the debris in the punch slot at time stamp t; e represents the natural constant. Represents the change amplitude of the debris state in the stamping groove from time t-1 to time t, and the change amplitude of the debris state is normalized by using natural constants, so as to characterize the change amplitude of the debris state based on a floating point number between 0 and 1, and the output parameters of the air compressor at time t are adjusted by using the change amplitude to obtain the output parameters of the air compressor at time t+1. Exemplarily, the output parameters of the air compressor include the instantaneous pressure of the compressed air at the nozzle outlet, the duration of a single blow, the interval time of periodic blows, the deflection angle of the blow direction relative to the debris accumulation surface, the lateral movement rate of the nozzle along the guide rail, etc.

[0050] S24, based on the output parameter, controlling the output state of the air compressor to the stamping groove.

[0051] In one embodiment of the present application, the output of the air compressor can be controlled based on the output parameters to adjust the output state of the air compressor to the stamping groove.

[0052] It can be seen from the above technical solution that the embodiment of the present application first determines the attribute data of the debris based on multiple monitoring images in the monitoring video, so as to improve the efficiency of detecting debris in the stamping slot. Then, the state of the debris in the stamping slot is determined based on the attribute data of the debris and the multimodal data acquired in real time. In this way, the debris state can be used to characterize the position change and cumulative trend change of the debris in the stamping slot, and the correlation between the output parameters of the air compressor and the debris state in the time dimension can be determined according to the debris state corresponding to different timestamps. In this way, the degree of influence of the output parameters of the air compressor on the debris state in the stamping slot can be determined in the time dimension and the space dimension, and finally the output parameters of the air compressor are updated according to the predicted debris state. In this way, the output state of the air compressor can be adjusted based on the data related to the debris in the stamping slot acquired in real time, thereby improving the cleaning efficiency and accuracy of the stamping slot.

[0053] See also Figure 3 , Figure 3 It is a functional module diagram of a punching slot cleaning system based on machine vision provided in one embodiment of the present application. The punching slot cleaning system 500 based on machine vision includes an electronic device 100, which is communicatively connected to an air compressor 200. The module / unit referred to in the present application refers to a series of computer-readable instruction segments that can be executed by a processor 13 and can perform fixed functions, which are stored in a memory 12. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0054] The electronic device 100 is used to obtain a monitoring video of the punching slot in real time, and the monitoring video includes at least one monitoring image.

[0055] The electronic device 100 is further configured to determine, based on a timestamp of acquiring the monitoring image, the multimodal data of the punching groove corresponding to the timestamp.

[0056] The electronic device 100 is further used to estimate the state of the stamping slot according to the monitoring image and the corresponding multimodal data to obtain the state of debris in the stamping slot.

[0057] The electronic device 100 is further used to determine the output parameters of the air compressor 200 according to the debris state.

[0058] The electronic device 100 is further used to control the output state of the air compressor 200 to the punching groove based on the output parameter.

[0059] In some embodiments, the electronic device 100 is also used to process the monitoring image according to a pre-trained target recognition model to obtain attribute data of the debris in the monitoring image; determine the cumulative trend of the debris in the monitoring image corresponding to the timestamp based on the attribute data of the debris in multiple monitoring images; and predict the state of the debris in the stamping groove based on the cumulative trend and the multimodal data.

[0060] In some embodiments, the electronic device 100 is also used to obtain historical images in historical surveillance videos, wherein the historical images correspond to label attribute data; input the historical images into a pre-constructed initial recognition model to obtain predicted attribute data corresponding to the historical images; determine a first loss value of the initial recognition model based on the predicted attribute data and the label attribute data; update the initial recognition model according to a back-propagation algorithm until the first loss value meets a preset condition, stop updating the initial recognition model, and obtain a target recognition model trained to a convergence state.

[0061] In some embodiments, the electronic device 100 is further configured to construct a first loss function according to the predicted attribute data and the label attribute data, and determine the first loss value according to the first loss function, wherein the first loss function satisfies the following relationship:

[0062] ; Wherein, Loss1 represents the first loss value of the initial recognition model; i is the index of the historical image, m is the number of the historical images; j is the index of the debris in the historical image indicated by the label attribute data, and n is the number of debris in the historical image indicated by the label attribute data; is the horizontal coordinate of the center point of the debris with index j in the i-th historical image indicated by the label attribute data, is the abscissa of the center point of the debris with index j in the i-th historical image indicated by the predicted attribute data; is the ordinate of the center point of the debris with index j in the i-th historical image indicated by the label attribute data, is the ordinate of the center point of the debris with index j in the i-th historical image indicated by the predicted attribute data; is the number of pixels occupied by debris with index j in the i-th historical image indicated by the label attribute data, is the number of pixels occupied by debris with index j in the i-th historical image indicated by the predicted attribute data; is the probability value that the debris with index j in the ith historical image indicated by the label attribute data belongs to the kth category, is the probability value that the debris with index j in the ith historical image indicated by the predicted attribute data belongs to the kth category; wherein k is the index of the category corresponding to the debris, and f is the number of categories corresponding to the debris.

[0063] In some embodiments, the electronic device 100 is also used to determine the movement distance and movement direction of the debris in the stamping groove corresponding to each timestamp based on the attribute data of the debris in any two adjacent monitoring images; determine the cumulative trend of the debris in the monitoring image corresponding to the timestamp based on the multimodal data corresponding to each timestamp and the movement distance and movement direction of the debris in the stamping groove corresponding to each timestamp; the cumulative trend is used to indicate the cumulative weight and cumulative range corresponding to each type of debris.

[0064] In some embodiments, the electronic device 100 is further used to determine the state data corresponding to the timestamp according to the cumulative trend and the multimodal data; and predict the state of the debris in the punching groove corresponding to the next timestamp according to the state data corresponding to the timestamp and the output parameter of the air compressor corresponding to the timestamp, including:

[0065] ;in, represents the state of the debris in the punching groove corresponding to the time stamp t+1; Represents the state data in the punching slot corresponding to the time stamp t; represents the output parameter of the air compressor at time t, A weight matrix representing the output parameters of the air compressor at timestamp t; Represents the error data at the preset timestamp t.

[0066] In some embodiments, the electronic device 100 determines the output parameter of the air compressor according to the debris state, including:

[0067] ;in, Represents the output parameters of the space compressor at timestamp t+1; Represents the output parameters of the air compressor at timestamp t; represents the state of the debris in the punch slot at the time stamp t-1; represents the state of debris in the punch groove at timestamp t; e represents a natural constant.

[0068] See also Figure 4, is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 100 includes a memory 12 and a processor 13. The memory 12 is used to store computer-readable instructions, and the processor 13 is used to execute the computer-readable instructions stored in the memory to implement a punching groove cleaning method based on machine vision described in any of the above embodiments.

[0069] In an embodiment of the present application, the electronic device 100 further includes a bus, a computer program stored in the memory 12 and executable on the processor 13, such as a punching groove cleaning program based on machine vision.

[0070] Figure 4 Only the electronic device 100 having the memory 12 and the processor 13 is shown, and those skilled in the art can understand that Figure 4 The structure shown does not constitute a limitation on the electronic device 100 , and the electronic device 100 may include fewer or more components than shown in the figure, or combine some components, or arrange the components differently.

[0071] Combination Figure 2 The memory 12 in the electronic device 100 stores a plurality of computer-readable instructions to implement the stamping slot cleaning method based on machine vision, and the processor 13 can execute the plurality of instructions to achieve: real-time acquisition of monitoring video of the stamping slot, the monitoring video including at least one monitoring image; determining multimodal data of the stamping slot corresponding to the timestamp according to the timestamp of acquiring the monitoring image; the multimodal data at least includes the weight of debris in the stamping slot and the airflow pressure of the air compressor; performing state estimation on the stamping slot according to the monitoring image and the corresponding multimodal data to obtain the state of debris in the stamping slot; determining the output parameters of the air compressor according to the state of debris; and controlling the output state of the air compressor to the stamping slot based on the output parameters.

[0072] Specifically, the specific implementation method of the processor 13 for the above instructions can refer to Figure 3 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0073] Those skilled in the art will appreciate that the schematic diagram is merely an example of the electronic device 100 and does not constitute a limitation on the electronic device 100. The electronic device 100 may be a bus-type structure or a star-type structure. The electronic device 100 may also include more or less other hardware or software than shown in the diagram, or a different arrangement of components. For example, the electronic device 100 may also include input and output devices, network access devices, etc.

[0074] It should be noted that the electronic device 100 is only an example, and other existing or future electronic products that are suitable for the present application should also be included in the protection scope of the present application and included here by reference.

[0075] Among them, the memory 12 includes at least one type of readable storage medium, and the readable storage medium can be non-volatile or volatile. The readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the electronic device 100, such as a mobile hard disk of the electronic device 100. In other embodiments, the memory 12 can also be an external storage device of the electronic device 100, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the electronic device 100. The memory 12 can not only be used to store application software and various types of data installed in the electronic device 100, such as a code of a stamping slot cleaning program based on machine vision, but also can be used to temporarily store data that has been output or is to be output.

[0076] In some embodiments, the processor 13 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The processor 13 is the control core (Control Unit) of the electronic device 100, and uses various interfaces and lines to connect various components of the entire electronic device 100, and executes or executes programs or modules stored in the memory 12 (for example, executing a machine vision-based stamping slot cleaning program, etc.), and calls data stored in the memory 12 to execute various functions of the electronic device 100 and process data.

[0077] The processor 13 executes the operating system of the electronic device 100 and various installed applications. The processor 13 executes the applications to implement the steps in the above-mentioned embodiments of the punching groove cleaning method based on machine vision, for example Figure 2 Steps shown.

[0078] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete the present application. The one or more modules / units may be a series of computer-readable instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the electronic device 100.

[0079] The above-mentioned integrated unit implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, and includes a number of instructions for enabling a computer device (which can be a personal computer, a computer device, or a network device, etc.) or a processor to execute a part of a machine vision-based stamping groove cleaning method described in each embodiment of the present application.

[0080] If the module / unit integrated in the electronic device 100 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also instruct the relevant hardware devices to complete through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented.

[0081] The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory and other memory, etc.

[0082] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0083] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. Figure 4Only one arrow is used in the figure, but it does not mean that there is only one bus or one type of bus. The bus is configured to realize the connection and communication between the memory 12 and at least one processor 13, etc.

[0084] An embodiment of the present application also provides a computer-readable storage medium (not shown), in which computer-readable storage medium is stored computer-readable instructions, and the computer-readable instructions are executed by a processor in an electronic device to implement the machine vision-based stamping groove cleaning method described in any of the above embodiments.

[0085] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0086] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0087] In addition, each functional module in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0088] In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the specification can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any specific order.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application has been described in detail with reference to the preferred embodiments, a person of ordinary skill in the art should understand that the technical solution of the present application may be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present application.

Claims

1. A stamping slot cleaning method based on machine vision, applied to an electronic device, wherein the electronic device is communicatively connected to an air compressor, characterized in that: The method comprises: Acquire a monitoring video of the punching slot in real time, wherein the monitoring video includes at least one monitoring image; Determining, according to a timestamp of acquiring the monitoring image, multimodal data of the punching slot corresponding to the timestamp; Performing state estimation on the punching slot according to the monitoring image and the corresponding multimodal data to obtain a state of debris in the punching slot; determining output parameters of the air compressor according to the state of the debris; Based on the output parameter, the output state of the air compressor to the punching groove is controlled.

2. The method for cleaning a stamping groove based on machine vision according to claim 1, characterized in that: The step of estimating the state of the punching slot according to the monitoring image and the corresponding multimodal data to obtain the state of debris in the punching slot includes: Processing the monitoring image according to a pre-trained target recognition model to obtain attribute data of debris in the monitoring image; Determining, based on the attribute data of the debris in the plurality of monitoring images, a cumulative trend of the debris in the monitoring images corresponding to the timestamp; A chip state in the punch slot is predicted based on the cumulative trend and the multimodal data.

3. The machine vision-based stamping groove cleaning method according to claim 2, characterized in that: The method further includes training the target recognition model, wherein the training the target recognition model includes: Acquire historical images in historical surveillance videos, wherein the historical images correspond to tag attribute data; Inputting the historical image into a pre-built initial recognition model to obtain predicted attribute data corresponding to the historical image; Determining a first loss value of the initial recognition model according to the predicted attribute data and the label attribute data; The initial recognition model is updated according to the back propagation algorithm until the first loss value meets a preset condition, and the updating of the initial recognition model is stopped to obtain a target recognition model trained to a convergent state.

4. The method for cleaning a stamping groove based on machine vision according to claim 3, characterized in that: Determining the first loss value of the initial recognition model according to the predicted attribute data and the label attribute data includes: A first loss function is constructed according to the predicted attribute data and the label attribute data, and the first loss value is determined according to the first loss function. The first loss function satisfies the following relationship: ； Wherein, Loss1 represents the first loss value of the initial recognition model; i is the index of the historical image, m is the number of the historical images; j is the index of the debris in the historical image indicated by the label attribute data, and n is the number of debris in the historical image indicated by the label attribute data; is the horizontal coordinate of the center point of the debris with index j in the i-th historical image indicated by the label attribute data, is the abscissa of the center point of the debris with index j in the i-th historical image indicated by the predicted attribute data; is the ordinate of the center point of the debris with index j in the i-th historical image indicated by the label attribute data, is the ordinate of the center point of the debris with index j in the i-th historical image indicated by the predicted attribute data; is the number of pixels occupied by debris with index j in the i-th historical image indicated by the label attribute data, is the number of pixels occupied by debris with index j in the i-th historical image indicated by the predicted attribute data; is the probability value that the debris with index j in the ith historical image indicated by the label attribute data belongs to the kth category, is the probability value that the debris with index j in the ith historical image indicated by the predicted attribute data belongs to the kth category; k is the index of the category corresponding to the debris, and f is the number of categories corresponding to the debris.

5. The method for cleaning a stamping groove based on machine vision according to claim 2, characterized in that: Determining the accumulation trend of the debris in the monitoring images corresponding to the timestamp according to the attribute data of the debris in the multiple monitoring images includes: Determine the moving distance and moving direction of the debris in the punching slot corresponding to each time stamp according to the attribute data of the debris in any two adjacent monitoring images; According to the multimodal data corresponding to each timestamp and the moving distance and moving direction of the debris in the stamping groove corresponding to each timestamp, the cumulative trend of the debris in the monitoring image corresponding to the timestamp is determined; the cumulative trend is used to indicate the cumulative weight and cumulative range corresponding to each type of debris.

6. The method for cleaning a stamping groove based on machine vision according to claim 2, characterized in that: The predicting of the chip state in the punching slot according to the cumulative trend and the multimodal data comprises: Determining state data corresponding to the timestamp according to the cumulative trend and the multimodal data; Predicting the state of the chips in the punching groove corresponding to the next time stamp according to the state data corresponding to the time stamp and the output parameter of the air compressor corresponding to the time stamp includes: ； in, represents the state of the debris in the punching groove corresponding to the time stamp t+1; Represents the state data in the punching slot corresponding to the time stamp t; represents the output parameter of the air compressor at time t, A weight matrix representing the output parameters of the air compressor at timestamp t; Represents the error data at the preset timestamp t.

7. The method for cleaning a stamping groove based on machine vision according to claim 1, characterized in that: Determining the output parameters of the air compressor according to the debris state includes: ； in, Represents the output parameters of the space compressor at timestamp t+1; Represents the output parameters of the air compressor at timestamp t; represents the state of the debris in the punch slot at the time stamp t-1; represents the state of debris in the punch groove at timestamp t; e represents a natural constant.

8. A punching slot cleaning system based on machine vision, characterized in that: The system includes an electronic device, the electronic device being communicatively connected to the air compressor; The electronic device is used to obtain a monitoring video of the punching slot in real time, wherein the monitoring video includes at least one monitoring image; The electronic device is also used to determine the multimodal data of the punching slot corresponding to the timestamp according to the timestamp of acquiring the monitoring image; The electronic device is further used to estimate the state of the punching slot according to the monitoring image and the corresponding multimodal data to obtain the state of debris in the punching slot; The electronic device is also used to determine the output parameters of the air compressor according to the state of the debris; The electronic device is also used to control the output state of the air compressor to the punching groove based on the output parameter.

9. The machine vision-based punching slot cleaning system according to claim 8, characterized in that: The electronic device is also used for: Processing the monitoring image according to a pre-trained target recognition model to obtain attribute data of debris in the monitoring image; Determining, based on the attribute data of the debris in the plurality of monitoring images, a cumulative trend of the debris in the monitoring images corresponding to the timestamp; A chip state in the punch slot is predicted based on the cumulative trend and the multimodal data.

10. The machine vision-based punching slot cleaning system according to claim 9, characterized in that: The electronic device is also used for: Acquire historical images in historical surveillance videos, wherein the historical images correspond to tag attribute data; Inputting the historical image into a pre-built initial recognition model to obtain predicted attribute data corresponding to the historical image; Determining a first loss value of the initial recognition model according to the predicted attribute data and the label attribute data; The initial recognition model is updated according to the back propagation algorithm until the first loss value meets a preset condition, and the updating of the initial recognition model is stopped to obtain a target recognition model trained to a convergent state.

Images