A vending machine material channel monitoring system based on machine vision

Abstract

The present invention provides a machine vision-based vending machine channel monitoring system, comprising a channel product data acquisition module, which utilizes a machine vision monitoring device to capture images of goods entering and exiting the vending machine channel; a channel product anomaly data acquisition module, which processes the channel product images based on a classification detection model to obtain anomaly images and channel profile data; a channel product anomaly data marking and processing module, which marks abnormal locations of the channel based on the anomaly images, generates an abnormal location distribution map, and processes channel anomalies based on the abnormal location distribution map; and a channel wear maintenance cycle prediction module, which scores channel profile data based on a preset channel wear scoring formula, generates a channel wear maintenance prediction model based on the scoring results, and uses the channel wear maintenance prediction model to perform regular maintenance on the channel. This invention improves the automation level of channel control.

Description

Technical Field

[0001] The present invention relates to the field of automated monitoring of vending machine equipment, and in particular to a vending machine material channel monitoring system based on machine vision. Background Art

[0002] Machine vision is the use of machines instead of human eyes to make measurements and judgments. It means that the captured target is converted into an image signal by a machine vision product, transmitted to a dedicated image processing system, and the morphological information of the captured target is obtained. According to the pixel distribution, brightness, color and other information, it is converted into a digital signal; the image system performs various operations on these signals to extract the characteristics of the target, and then controls the operation of the equipment on site according to the judgment results.

[0003] Machine vision systems can quickly acquire large amounts of information, easily process it automatically, and integrate it with design and process control information. Machine vision is often used to replace human vision in hazardous work environments where manual labor is unsuitable or where human vision is insufficient.

[0004] At present, there are few monitoring devices on the material channels of vending machines on the market. The types of goods sold are diverse, and the delivery method of the material channel is prone to abnormalities. Once the material channel is out of stock, abnormal delivery of goods, and other faults occur during use, it will affect the efficiency of use. Summary of the Invention

[0005] The present invention provides a vending machine material channel monitoring system based on machine vision. By utilizing a machine vision device to obtain image data inside the material channel and analyzing and processing abnormal images of the material channel, it realizes automatic monitoring of the process of goods entering and exiting the material channel, thereby improving the degree of automation of the control of goods entering and exiting the material channel.

[0006] The present invention provides a vending machine material channel monitoring system based on machine vision, comprising:

[0007] The material channel product data acquisition module is used to collect and obtain images of goods entering and leaving the material channel of the vending machine using a machine vision monitoring device;

[0008] The abnormal data acquisition module for goods in the material channel is used to process the images of goods entering and exiting the material channel based on the classification detection model to obtain abnormal images of goods entering and exiting the material channel and the material channel profile data;

[0009] The abnormal data marking and processing module for goods in the material channel is used to mark the abnormal positions of goods in and out of the material channel according to the abnormal images of goods in and out of the material channel, generate an abnormal position distribution positioning map, and process the material channel abnormalities according to the abnormal position distribution positioning map;

[0010] The material channel wear maintenance cycle prediction module is used to score the material channel profile data based on a preset material channel wear scoring formula, generate a material channel wear maintenance prediction model based on the scoring results, and use the material channel wear maintenance prediction model to perform regular maintenance on the material channel.

[0011] Furthermore, the machine vision monitoring device includes a laser camera and several groups of charge-coupled device cameras. The laser camera is used to collect images of the side walls and bottom of the material channel; the charge-coupled device camera includes a first camera group and a second camera group; the first camera group is arranged at the material channel entrance of the vending machine, and is used to capture images of goods at the material channel entrance; the second camera group is arranged in the material channel of the vending machine, and is used to capture images of the goods shipping process in the material channel.

[0012] Furthermore, the material channel goods data acquisition module also includes an image noise reduction processing evaluation unit and an abnormality type judgment unit;

[0013] The image noise reduction evaluation unit is used to evaluate the noise reduction effect of the goods inlet and outlet channel images after noise reduction processing, and to screen the images and determine the abnormality type based on the evaluation results; including:

[0014] Based on a preset subjective evaluation method and subjective evaluation rules, the image effect is evaluated to obtain a subjective evaluation score; the subjective evaluation method is for a person to view the image; the subjective evaluation rules are based on the image noise attenuation status and whether the image edge is blurred;

[0015] Based on a preset objective evaluation method and objective evaluation indicators, the image effect is evaluated to obtain an objective evaluation score; the objective evaluation method is to calculate the signal-to-noise ratio of the image, that is, to calculate the variance ratio of the signal to the noise; the objective evaluation indicator is a preset objective evaluation indicator based on the change in image variance and the change in the image signal-to-noise ratio value;

[0016] The subjective evaluation score and the objective evaluation score are weighted and summed to obtain a comprehensive evaluation score; if the comprehensive evaluation score is greater than or equal to a preset comprehensive evaluation score threshold, the image corresponding to the comprehensive evaluation score is retained; if the comprehensive evaluation score is less than the preset comprehensive evaluation score threshold, the image corresponding to the comprehensive evaluation score is discarded;

[0017] The abnormality category judgment unit is used to judge the abnormality of shortage and jam of the material channel; after performing noise reduction processing on the image of the goods at the feed channel position, if the subjective evaluation score is the preset maximum value, it is judged that the material channel is out of stock and abnormal; if the subjective evaluation score is less than the preset maximum value, it is judged that the material channel is jammed.

[0018] Furthermore, the abnormal data acquisition module for goods in the material channel includes a data sample set classification unit, a classification detection training unit and a classification detection implementation unit;

[0019] A data sample set classification unit is used to divide the images of goods entering and exiting the material channel into a training data sample set and a detection data sample set;

[0020] The classification detection training unit is used to extract features from the training data sample set based on the preset classification detection model using a fusion feature extraction method and to train based on the SVM classifier;

[0021] The classification detection implementation unit is used to perform classification detection on the detection data sample set based on the classification detection model to obtain abnormal images of the goods in and out of the material channel.

[0022] Furthermore, the fusion feature extraction method includes using a directional gradient histogram feature extraction method to extract the edge gradient feature vector of the local target in the image data, using a local binary pattern feature extraction method to extract the image texture feature vector, and serially fusing the edge gradient feature vector of the local target with the image texture feature vector.

[0023] Furthermore, the abnormal data acquisition module for goods in the material channel also includes a profile data acquisition unit and a profile data processing unit;

[0024] A profile data acquisition unit, used for collecting profile data of the side wall and bottom images of the material channel using a profilometer;

[0025] The profile data processing unit is used to process abnormal data points that do not meet the rules in a preset manner to obtain the material channel profile data; the preset method is: first delete the abnormal points, then use the data points before and after, and take the average for interpolation according to the rule that the data curve shows an overall downward trend.

[0026] Furthermore, the material channel goods abnormal data marking processing module includes: an image marking classification unit, an image marking operation unit and a material channel abnormality processing unit;

[0027] An image marking and classification unit is used to classify abnormal images of the goods inlet and outlet channels into two categories based on the clarity of the abnormal positions in the abnormal images of the goods inlet and outlet channels; images with a clarity greater than or equal to a preset clarity threshold are classified as clear images, and images with a clarity less than the preset clarity threshold are classified as blurred images;

[0028] An image marking operation unit is used to mark the specific contours of the abnormality occurrence location in the clear image using a mask marking method, and generate a PNG format image file containing the mask trajectory information of the abnormality occurrence location; mark the detection area of the abnormality occurrence location in the blurred image using a detection frame marking method, and generate a JSON format text file containing the coordinates of the detection frame vertices and the abnormality location information; and aggregate the PNG format image file and the JSON format text file to generate an abnormality location distribution map;

[0029] The material channel abnormality processing unit is used to process abnormal situations in the abnormal position distribution positioning map based on a preset processing solution.

[0030] Furthermore, the material channel wear maintenance cycle prediction module includes a scoring result generation unit, a cycle prediction model generation unit and a cycle prediction implementation unit;

[0031] A scoring result generating unit is used to calculate the index parameter values for material channel damage scoring based on the profile data; the index parameter values include normal value, wear area, vertical wear value and side wear value; the index parameters are substituted into a preset scoring formula to obtain a scoring result;

[0032] The cycle prediction model generation unit is used to calculate the monthly average of the scoring data within a cycle based on the scoring results, construct a time series, and then use the least squares method to fit it to generate a cycle prediction model;

[0033] The cycle prediction implementation unit is used to predict the maintenance cycle according to the cycle prediction model to obtain the maintenance cycle.

[0034] Furthermore, it also includes a remote automatic processing module for automatically handling abnormal situations of some stuck goods; the remote automatic processing module includes a stuck goods risk determination unit, an abnormality warning processing unit and a material channel automatic adjustment unit;

[0035] The cargo jam risk determination unit is used to set a corresponding cargo jam risk value based on historical abnormal images of cargo inlet and outlet channels, and to set a matching table between abnormal images and cargo jam risk values; to query the matching table to determine the cargo jam risk value corresponding to the current abnormal image of the cargo inlet and outlet channels; to obtain all the cargo jam risk values, to sum all the cargo jam risk values, and to obtain a cumulative cargo jam risk value; and to determine the cargo jam risk level based on the magnitude of the cumulative cargo jam risk value; the cargo jam risk levels include high risk, medium risk, and low risk;

[0036] The early warning processing command sending unit is used to generate and send early warning processing commands to the material channel automatic adjustment unit based on the medium risk and low risk levels.

[0037] The material channel automatic adjustment unit is used to automatically handle the jamming of goods in the material channel; the material channel automatic adjustment unit includes a material channel width fine-tuning device; when receiving a control command, the material channel width fine-tuning device adjusts the material channel width; when receiving the first control command corresponding to the low-risk warning, the material channel width fine-tuning device adjusts the material channel width according to the preset first adjustment value; when receiving the second control command corresponding to the medium-risk warning, the material channel width fine-tuning device adjusts the material channel width according to the preset second adjustment value.

[0038] Furthermore, it also includes a material channel maintenance effect detection unit for detecting the effect of the material channel maintenance;

[0039] The material channel maintenance effect detection unit includes a pre-maintenance test unit, a cleaning and maintenance unit, and a post-maintenance detection unit;

[0040] The pre-maintenance test unit is used to inject a test fluid into the material channel and calculate the total resistance of the test fluid flowing in the material channel before the test based on the flow resistance and shape resistance of the test fluid;

[0041] The cleaning and maintenance unit is used to clean and maintain the inside of the material channel by sweeping and dusting, fine grinding and leveling, dust removal and air drying and spraying;

[0042] The post-maintenance detection unit is used to inject a test fluid into the material channel after cleaning and maintenance, and calculate the total resistance of the test fluid flowing in the material channel after maintenance by measuring the flow resistance and shape resistance of the test fluid.

[0043] Obtain the initial total resistance of the test fluid flowing in the material channel when the material channel is not used after leaving the factory, calculate the difference between the total resistance before measurement and the initial total resistance, and obtain a first difference; calculate the difference between the total resistance after maintenance and the initial total resistance, and obtain a second difference; calculate the difference between the first difference and the second difference, and obtain a third difference; when the third difference is less than or equal to the preset difference threshold, it is determined that the maintenance effect is qualified; otherwise, continue the cleaning and maintenance work until the maintenance effect is determined to be qualified.

[0044] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0045] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0047] Figure 1 This is a schematic structural diagram of a vending machine material channel monitoring system based on machine vision according to the present invention;

[0048] Figure 2 This is a structural diagram of a module for acquiring abnormal data on goods in a vending machine channel monitoring system based on machine vision according to the present invention;

[0049] Figure 3This is a structural schematic diagram of a module for marking and processing abnormal data of goods in a vending machine channel monitoring system based on machine vision according to the present invention. DETAILED DESCRIPTION

[0050] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0051] A vending machine material channel monitoring system based on machine vision, such as Figure 1 As shown, including:

[0052] The material channel product data acquisition module is used to collect and obtain images of goods entering and leaving the material channel of the vending machine using a machine vision monitoring device;

[0053] The abnormal data acquisition module for goods in the material channel is used to process the images of goods entering and exiting the material channel based on the classification detection model to obtain abnormal images of goods entering and exiting the material channel and the material channel profile data;

[0054] The abnormal data marking and processing module for goods in the material channel is used to mark the abnormal positions of goods in and out of the material channel according to the abnormal images of goods in and out of the material channel, generate an abnormal position distribution positioning map, and process the material channel abnormalities according to the abnormal position distribution positioning map;

[0055] The material channel wear maintenance cycle prediction module is used to score the material channel profile data based on a preset material channel wear scoring formula, generate a material channel wear maintenance prediction model based on the scoring results, and use the material channel wear maintenance prediction model to perform regular maintenance on the material channel.

[0056] The working principle of the above technical solution is as follows: using machine vision equipment to collect image data of goods entering and exiting the material channel is the basis for monitoring the material channel. This data is then processed to obtain abnormal images and material channel profile data. Abnormal data is then marked and processed, and the profile data is scored for wear. Based on the scoring results, the material channel is regularly maintained. Specifically, the material channel product data acquisition module is used to use machine vision monitoring equipment to collect images of goods entering and exiting the material channel of the vending machine;

[0057] The abnormal data acquisition module for goods in the material channel is used to process the images of goods entering and exiting the material channel based on the classification detection model to obtain abnormal images of goods entering and exiting the material channel and the material channel profile data;

[0058] The abnormal data marking and processing module for goods in the material channel is used to mark the abnormal positions of goods in and out of the material channel according to the abnormal images of goods in and out of the material channel, generate an abnormal position distribution positioning map, and process the material channel abnormalities according to the abnormal position distribution positioning map;

[0059] The material channel wear maintenance cycle prediction module is used to score the material channel profile data based on a preset material channel wear scoring formula, generate a material channel wear maintenance prediction model based on the scoring results, and use the material channel wear maintenance prediction model to perform regular maintenance on the material channel.

[0060] The beneficial effects of the above technical solution are: by adopting the solution provided in this embodiment, by using a machine vision device to obtain internal image data of the material channel, and by analyzing and processing abnormal images of the material channel, automatic monitoring of the process of goods entering and exiting the material channel is achieved, thereby improving the degree of automation of the control of goods entering and exiting the material channel.

[0061] In one embodiment, the machine vision monitoring device includes a laser camera and several groups of charge-coupled device cameras. The laser camera is used to collect images of the side walls and bottom of the material channel; the charge-coupled device camera includes a first camera group and a second camera group; the first camera group is arranged at the inlet of the vending machine material channel, and is used to capture images of goods at the inlet; the second camera group is arranged in the vending machine material channel, and is used to capture images of the goods shipping process in the material channel.

[0062] The working principle of the above technical solution is as follows: In a machine vision system, it is crucial to obtain a high-quality, processable image. In the process of goods entering and exiting the material channel, abnormal situations that are prone to occur mainly include out-of-stock abnormalities and jammed goods abnormalities. For these two situations, it is necessary to set a camera group at the corresponding position in the material channel to collect image data. Specifically: the visual monitoring device includes a laser camera and several groups of charge-coupled device cameras. The laser camera is used to collect images of the side walls and bottom of the material channel; the charge-coupled device camera includes a first camera group and a second camera group; the first camera group is set at the inlet of the vending machine material channel to capture images of goods at the inlet; the second camera group is set in the vending machine material channel to capture images of the goods shipping process in the material channel.

[0063] The beneficial effect of the above technical solution is: by adopting the solution provided by this embodiment, by setting camera groups at different positions in the material channel, image data can be collected in a targeted manner, providing valuable image data for material channel monitoring.

[0064] In one embodiment, the material channel product data acquisition module further includes an image noise reduction processing evaluation unit and an abnormality type judgment unit;

[0065] The image noise reduction evaluation unit is used to evaluate the noise reduction effect of the goods inlet and outlet channel images after noise reduction processing, and to screen the images and determine the abnormality type based on the evaluation results; including:

[0066] Based on a preset subjective evaluation method and subjective evaluation rules, the image effect is evaluated to obtain a subjective evaluation score; the subjective evaluation method is for a person to view the image; the subjective evaluation rules are based on the image noise attenuation status and whether the image edge is blurred;

[0067] Based on a preset objective evaluation method and objective evaluation indicators, the image effect is evaluated to obtain an objective evaluation score; the objective evaluation method is to calculate the signal-to-noise ratio of the image, that is, to calculate the variance ratio of the signal to the noise; the objective evaluation indicator is a preset objective evaluation indicator based on the change in image variance and the change in the image signal-to-noise ratio value;

[0068] The subjective evaluation score and the objective evaluation score are weighted and summed to obtain a comprehensive evaluation score; if the comprehensive evaluation score is greater than or equal to a preset comprehensive evaluation score threshold, the image corresponding to the comprehensive evaluation score is retained; if the comprehensive evaluation score is less than the preset comprehensive evaluation score threshold, the image corresponding to the comprehensive evaluation score is discarded;

[0069] The abnormality category judgment unit is used to judge the abnormality of shortage and jam of the material channel; after performing noise reduction processing on the image of the goods at the feed channel position, if the subjective evaluation score is the preset maximum value, it is judged that the material channel is out of stock and abnormal; if the subjective evaluation score is less than the preset maximum value, it is judged that the material channel is jammed.

[0070] The working principle of the above technical solution is as follows: image quality has a great influence on the recognition effect in image recognition processing. Since there are many uncertain factors such as image acquisition, transmission, cutting and processing, as well as ambient lighting, which can cause image noise, in order to ensure qualified image quality, noise reduction processing is required and the effect of noise reduction processing is evaluated; at the same time, the abnormality type is determined based on the image abnormality, such as whether the abnormality of the material channel is out of stock or stuck; specifically, the image noise reduction processing evaluation unit is used to evaluate the noise reduction processing effect after performing noise reduction processing on the images of goods entering and leaving the material channel, and screen the images and determine the abnormality type based on the evaluation results; including:

[0071] Based on a preset subjective evaluation method and subjective evaluation rules, the image effect is evaluated to obtain a subjective evaluation score; the subjective evaluation method is for a person to view the image; the subjective evaluation rules are based on the image noise attenuation status and whether the image edge is blurred;

[0072] Based on a preset objective evaluation method and objective evaluation indicators, the image effect is evaluated to obtain an objective evaluation score; the objective evaluation method is to calculate the signal-to-noise ratio of the image, that is, to calculate the variance ratio of the signal to the noise; the objective evaluation indicator is a preset objective evaluation indicator based on the change in image variance and the change in the image signal-to-noise ratio value;

[0073] The subjective evaluation score and the objective evaluation score are weighted and summed to obtain a comprehensive evaluation score; if the comprehensive evaluation score is greater than or equal to a preset comprehensive evaluation score threshold, the image corresponding to the comprehensive evaluation score is retained; if the comprehensive evaluation score is less than the preset comprehensive evaluation score threshold, the image corresponding to the comprehensive evaluation score is discarded;

[0074] The abnormality category judgment unit is used to judge the abnormality of shortage and jam of the material channel; after performing noise reduction processing on the image of the goods at the feed channel position, if the subjective evaluation score is the preset maximum value, it is judged that the material channel is out of stock and abnormal; if the subjective evaluation score is less than the preset maximum value, it is judged that the material channel is jammed.

[0075] The beneficial effect of the above technical solution is: by adopting the solution provided by this embodiment, by evaluating the effect of image noise reduction processing, it is possible to ensure that high-quality images that meet the requirements are retained, thereby ensuring the effect of subsequent image recognition processing.

[0076] In one embodiment, Figure 2 As shown, the abnormal data acquisition module for the material channel goods includes a data sample set classification unit, a classification detection training unit and a classification detection implementation unit;

[0077] A data sample set classification unit is used to divide the images of goods entering and exiting the material channel into a training data sample set and a detection data sample set;

[0078] The classification detection training unit is used to extract features from the training data sample set based on the preset classification detection model using a fusion feature extraction method and to train based on the SVM classifier;

[0079] The classification detection implementation unit is used to perform classification detection on the detection data sample set based on the classification detection model to obtain abnormal images of the goods in and out of the material channel.

[0080] The working principle of the above technical solution is as follows: in order to detect abnormal images, feature extraction and classification processing are required for the material channel image data to ensure that abnormal images are sorted out; specifically, the material channel goods abnormal data acquisition module includes a data sample set classification unit, a classification detection training unit, and a classification detection implementation unit;

[0081] A data sample set classification unit is used to divide the images of goods entering and exiting the material channel into a training data sample set and a detection data sample set;

[0082] The classification detection training unit is used to extract features from the training data sample set based on the preset classification detection model using a fusion feature extraction method and to train based on the SVM classifier;

[0083] The classification detection implementation unit is used to perform classification detection on the detection data sample set based on the classification detection model to obtain abnormal images of the goods in and out of the material channel.

[0084] The beneficial effect of the above technical solution is: by adopting the solution provided by this embodiment, abnormal images of goods in and out of the material channel can be effectively identified through feature extraction and classification recognition of the image, thereby improving the quality of image analysis and processing.

[0085] In one embodiment, the fusion feature extraction method includes using a directional gradient histogram feature extraction method to extract the edge gradient feature vector of the local target in the image data, using a local binary pattern feature extraction method to extract the image texture feature vector, and serially fusing the edge gradient feature vector of the local target with the image texture feature vector.

[0086] The working principle of the above technical solution is: a good feature can better describe the target in an appropriate scenario. The directional gradient histogram feature extraction method can well describe the edge features of the image by calculating the gradient histogram of the local image; the local binary pattern feature extraction method can effectively extract the texture features of the image; combining the two feature extraction methods can improve the extraction effect; specifically: pedestrian detection has been developed for many years. The fusion feature extraction method includes using the directional gradient histogram feature extraction method to extract the edge gradient feature vector of the local target in the image data, and using the local binary pattern feature extraction method to extract the image texture feature vector, and serially fusing the edge gradient feature vector of the local target with the image texture feature vector.

[0087] The beneficial effects of the above technical solution are: by adopting the solution provided in this embodiment, the directional gradient histogram feature extraction method can better extract the edge gradient features of local targets, and the local binary pattern feature extraction method can better describe the image texture features. Combining the two methods can improve the accuracy of image detection.

[0088] In one embodiment, the module for acquiring abnormal data of goods in the material channel further includes a profile data acquisition unit and a profile data processing unit;

[0089] A profile data acquisition unit, used for collecting profile data of the side wall and bottom images of the material channel using a profilometer;

[0090] The profile data processing unit is used to process abnormal data points that do not meet the rules in a preset manner to obtain the material channel profile data; the preset method is: first delete the abnormal points, then use the data points before and after, and take the average for interpolation according to the rule that the data curve shows an overall downward trend.

[0091] The working principle of the above technical solution is as follows: according to the profile score calculation method, the longer the material channel is worn and the more severe the wear, the lower the profile score should be, and the profile score tends to decline. This requires processing data points that do not meet the rules to avoid affecting the curve fitting effect. The abnormal data acquisition module for material channel products also includes a profile data acquisition unit and a profile data processing unit.

[0092] A profile data acquisition unit, used for collecting profile data of the side wall and bottom images of the material channel using a profilometer;

[0093] The profile data processing unit is used to process abnormal data points that do not meet the rules in a preset manner to obtain the material channel profile data; the preset method is: first delete the abnormal points, then use the data points before and after, and take the average for interpolation according to the rule that the data curve shows an overall downward trend.

[0094] Template matching can be applied to the field of object position and direction determination. Using a template image to describe the searched object is to find the template position in the image. The template needs to be moved to all positions in the image and the similarity between that position and all the template poses needs to be calculated. Similarity here is a broad concept that can be defined in different ways, such as based on grayscale values or edge similarity. The movement of the object's position and the change in direction can be represented by translation and rotation matrices, respectively, which are called rigid transformations. This embodiment involves the occurrence of goods jamming due to the translation of the position of the goods, and uses image matching and recognition based on the grayscale values of the template and the image.

[0095]

[0096] In the above formula, T ssd (i, j) represents the sum of the squares of all differences between the grayscale values of the pixels in the template and the grayscale values of the corresponding image points, p(i, j) represents the grayscale function of the template image, p(α, β) represents the image of the region of interest in the template image, and u(i+α, j+β) represents the grayscale value of the image area covered by the current position of the template; in order to perform template matching, the template is moved to all positions in the image and the similarity measure of the template at that position is calculated. The similarity measure has a value at each point in the matching process, and the sum of the squares of all differences between the grayscale values of the pixels in the template and the grayscale values of the corresponding image points is calculated. If T ssd If the value of (i, j) is zero, it means that the template and the image are the same. If they are different, the value will be greater than zero, and the larger the value, the greater the difference between the template and the image.

[0097] The beneficial effects of the above technical solution are: by adopting the solution provided in this embodiment, by obtaining and processing the profile data, and by matching and identifying the image based on the grayscale value of the template and the image, the speed of image recognition and matching can be improved, and the accuracy of the material channel profile data can be ensured.

[0098] In one embodiment, Figure 3 As shown, the material channel goods abnormal data marking processing module includes: an image marking classification unit, an image marking operation unit and a material channel abnormality processing unit;

[0099] An image marking and classification unit is used to classify abnormal images of the goods inlet and outlet channels into two categories based on the clarity of the abnormal positions in the abnormal images of the goods inlet and outlet channels; images with a clarity greater than or equal to a preset clarity threshold are classified as clear images, and images with a clarity less than the preset clarity threshold are classified as blurred images;

[0100] An image marking operation unit is used to mark the specific contours of the abnormality occurrence location in the clear image using a mask marking method, and generate a PNG format image file containing the mask trajectory information of the abnormality occurrence location; mark the detection area of the abnormality occurrence location in the blurred image using a detection frame marking method, and generate a JSON format text file containing the coordinates of the detection frame vertices and the abnormality location information; and aggregate the PNG format image file and the JSON format text file to generate an abnormality location distribution map;

[0101] The material channel abnormality processing unit is used to process abnormal situations in the abnormal position distribution positioning map based on a preset processing solution.

[0102] The working principle of the above technical solution is as follows: the image standards mainly include mask marking method and detection frame marking method; the mask marking method mainly marks the specific outline of the abnormal position; the detection frame marking method mainly marks the abnormal area and abnormal type information, and through image marking, it can provide a basis for further image analysis; specifically, the image marking classification unit is used to classify the abnormal images of the goods inlet and outlet channels into two categories based on the clarity of the abnormal position in the abnormal images of the goods inlet and outlet channels; images with clarity greater than or equal to the preset clarity threshold are classified as clear images, and images with clarity less than the preset clarity threshold are classified as blurred images;

[0103] An image marking operation unit is used to mark the specific contours of the abnormality occurrence location in the clear image using a mask marking method, and generate a PNG format image file containing the mask trajectory information of the abnormality occurrence location; mark the detection area of the abnormality occurrence location in the blurred image using a detection frame marking method, and generate a JSON format text file containing the coordinates of the detection frame vertices and the abnormality location information; and aggregate the PNG format image file and the JSON format text file to generate an abnormality location distribution map;

[0104] The material channel abnormality processing unit is used to process abnormal situations in the abnormal position distribution positioning map based on a preset processing solution.

[0105] The beneficial effect of the above technical solution is that the solution provided by this embodiment helps to clearly find abnormal locations and perform targeted processing through marking and storing images.

[0106] In one embodiment, the material channel wear maintenance cycle prediction module includes a scoring result generation unit, a cycle prediction model generation unit, and a cycle prediction implementation unit;

[0107] A scoring result generating unit is used to calculate the index parameter values for material channel damage scoring based on the profile data; the index parameter values include normal value, wear area, vertical wear value and side wear value; the index parameters are substituted into a preset scoring formula to obtain a scoring result;

[0108] The cycle prediction model generation unit is used to calculate the monthly average of the scoring data within a cycle based on the scoring results, construct a time series, and then use the least squares method to fit it to generate a cycle prediction model;

[0109] The cycle prediction implementation unit is used to predict the maintenance cycle according to the cycle prediction model to obtain the maintenance cycle.

[0110] The working principle of the above technical solution is as follows: the material channel loss score needs to be calculated using indicator parameters, and then a cycle prediction model is generated based on the score structure. The maintenance cycle is predicted using the cycle prediction model. Specifically, the material channel wear maintenance cycle prediction module includes a scoring result generation unit, a cycle prediction model generation unit, and a cycle prediction implementation unit.

[0111] A scoring result generating unit is used to calculate the index parameter values for material channel damage scoring based on the profile data; the index parameter values include normal value, wear area, vertical wear value and side wear value; the index parameters are substituted into a preset scoring formula to obtain a scoring result;

[0112] The cycle prediction model generation unit is used to calculate the monthly average of the scoring data within a cycle based on the scoring results, construct a time series, and then use the least squares method to fit it to generate a cycle prediction model;

[0113] The cycle prediction implementation unit is used to predict maintenance cycles based on the cycle prediction model and obtain the maintenance cycle. For example, based on the scoring structure data of the material channel for the first five months of use, the average score for these five months is calculated, and then the average score is used in the cycle prediction formula. The calculated time value corresponding to the profile score is then combined with the score value required for maintenance to solve for the corresponding time value. The difference between the two times is the time point for the next maintenance.

[0114] The beneficial effect of the above technical solution is: by adopting the solution provided by this embodiment, by calculating the maintenance cycle, a maintenance plan can be formulated in a planned manner, thereby improving the maintenance effect of the material channel and improving the utilization efficiency of the vending machine.

[0115] In one embodiment, a remote automatic processing module is further included for automatically processing some abnormal situations of cargo jams; the remote automatic processing module includes a cargo jam risk determination unit, an abnormality warning processing unit, and a material channel automatic adjustment unit;

[0116] The cargo jam risk determination unit is used to set a corresponding cargo jam risk value based on historical abnormal images of cargo inlet and outlet channels, and to set a matching table between abnormal images and cargo jam risk values; to query the matching table to determine the cargo jam risk value corresponding to the current abnormal image of the cargo inlet and outlet channels; to obtain all the cargo jam risk values, to sum all the cargo jam risk values, and to obtain a cumulative cargo jam risk value; and to determine the cargo jam risk level based on the magnitude of the cumulative cargo jam risk value; the cargo jam risk levels include high risk, medium risk, and low risk;

[0117] The early warning processing command sending unit is used to generate and send early warning processing commands to the material channel automatic adjustment unit based on the medium risk and low risk levels.

[0118] The material channel automatic adjustment unit is used to automatically handle the jamming of goods in the material channel; the material channel automatic adjustment unit includes a material channel width fine-tuning device; when receiving a control command, the material channel width fine-tuning device adjusts the material channel width; when receiving the first control command corresponding to the low-risk warning, the material channel width fine-tuning device adjusts the material channel width according to the preset first adjustment value; when receiving the second control command corresponding to the medium-risk warning, the material channel width fine-tuning device adjusts the material channel width according to the preset second adjustment value.

[0119] The working principle of the above technical solution is as follows: through the analysis of abnormal situations, it is necessary to formulate corresponding risk levels and handle them according to the level. For abnormal situations with medium and low risk levels, remote automatic processing and adjustment can improve the efficiency of vending machines. Due to the different situations of product jams, it is necessary to set corresponding risk values for different levels of product jams to more carefully understand the abnormal situations. Specifically, the remote automatic processing module is used to automatically handle some abnormal situations of product jams. The remote automatic processing module includes a product jam risk determination unit, an abnormality warning processing unit, and an automatic material channel adjustment unit.

[0120] The cargo jam risk determination unit is used to set a corresponding cargo jam risk value based on historical abnormal images of cargo inlet and outlet channels, and to set a matching table between abnormal images and cargo jam risk values; to query the matching table to determine the cargo jam risk value corresponding to the current abnormal image of the cargo inlet and outlet channels; to obtain all the cargo jam risk values, to sum all the cargo jam risk values, and to obtain a cumulative cargo jam risk value; and to determine the cargo jam risk level based on the magnitude of the cumulative cargo jam risk value; the cargo jam risk levels include high risk, medium risk, and low risk;

[0121] The early warning processing command sending unit is used to generate and send early warning processing commands to the material channel automatic adjustment unit based on the medium risk and low risk levels.

[0122] The material channel automatic adjustment unit is used to automatically handle the jamming of goods in the material channel; the material channel automatic adjustment unit includes a material channel width fine-tuning device; when receiving a control command, the material channel width fine-tuning device adjusts the material channel width; when receiving the first control command corresponding to the low-risk warning, the material channel width fine-tuning device adjusts the material channel width according to the preset first adjustment value; when receiving the second control command corresponding to the medium-risk warning, the material channel width fine-tuning device adjusts the material channel width according to the preset second adjustment value.

[0123] The beneficial effects of the above technical solution are: by adopting the solution provided by this embodiment, by setting up a remote automatic processing module and setting corresponding risk levels, the situation of abnormal stuck goods can be grasped in more detail, and abnormalities of medium and low risk levels can be remotely and automatically processed, thereby improving the degree of automation of abnormal handling of vending machines, and can promptly resolve minor material channel abnormalities, saving time and costs for manual maintenance, and improving the utilization efficiency of vending machines.

[0124] In one embodiment, a material channel maintenance effect detection unit is further included, which is used to detect the effect of the material channel maintenance;

[0125] The material channel maintenance effect detection unit includes a pre-maintenance test unit, a cleaning and maintenance unit, and a post-maintenance detection unit;

[0126] The pre-maintenance test unit is used to inject a test fluid into the material channel and calculate the total resistance of the test fluid flowing in the material channel before the test based on the flow resistance and shape resistance of the test fluid;

[0127] The cleaning and maintenance unit is used to clean and maintain the inside of the material channel by sweeping and dusting, fine grinding and leveling, dust removal and air drying and spraying;

[0128] The post-maintenance detection unit is used to inject a test fluid into the material channel after cleaning and maintenance, and calculate the total resistance of the test fluid flowing in the material channel after maintenance by measuring the flow resistance and shape resistance of the test fluid.

[0129] Obtain the initial total resistance of the test fluid flowing in the material channel when the material channel is not used after leaving the factory, calculate the difference between the total resistance before measurement and the initial total resistance, and obtain a first difference; calculate the difference between the total resistance after maintenance and the initial total resistance, and obtain a second difference; calculate the difference between the first difference and the second difference, and obtain a third difference; when the third difference is less than or equal to the preset difference threshold, it is determined that the maintenance effect is qualified; otherwise, continue the cleaning and maintenance work until the maintenance effect is determined to be qualified.

[0130] The working principle of the above technical solution is that the effect of material channel maintenance is related to the efficiency of material channel use. It is necessary to evaluate the maintenance effect to ensure the use of the material channel. Specifically, the material channel maintenance effect detection unit includes a pre-maintenance test unit, a cleaning and maintenance unit, and a post-maintenance detection unit.

[0131] The pre-maintenance test unit is used to inject a test fluid into the material channel and calculate the total resistance of the test fluid flowing in the material channel before the test based on the flow resistance and shape resistance of the test fluid;

[0132] The cleaning and maintenance unit is used to clean and maintain the inside of the material channel by sweeping and dusting, fine grinding and leveling, dust removal and air drying and spraying;

[0133] The post-maintenance detection unit is used to inject a test fluid into the material channel after cleaning and maintenance, and calculate the total resistance of the test fluid flowing in the material channel after maintenance by measuring the flow resistance and shape resistance of the test fluid.

[0134] Obtain the initial total resistance of the test fluid flowing in the material channel when the material channel is not used after leaving the factory, calculate the difference between the total resistance before measurement and the initial total resistance, and obtain a first difference; calculate the difference between the total resistance after maintenance and the initial total resistance, and obtain a second difference; calculate the difference between the first difference and the second difference, and obtain a third difference; when the third difference is less than or equal to the preset difference threshold, it is determined that the maintenance effect is qualified; otherwise, continue the cleaning and maintenance work until the maintenance effect is determined to be qualified.

[0135] The beneficial effect of the above technical solution is: by adopting the solution provided in this embodiment and evaluating the maintenance effect of the material channel, the quality of the material channel cleaning and maintenance can be effectively improved, and the service life of the material channel can be increased.

[0136] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A vending machine material channel monitoring system based on machine vision, characterized in that: include: The material channel product data acquisition module is used to collect and obtain images of goods entering and leaving the material channel of the vending machine using a machine vision monitoring device; The abnormal data acquisition module for goods in the material channel is used to process the images of goods entering and exiting the material channel based on the classification detection model, and obtain abnormal images of goods entering and exiting the material channel and the material channel profile data; specifically: Divide the images of goods entering and exiting the material channel into a training data sample set and a detection data sample set; For the training data sample set, based on the preset classification detection model, a fusion feature extraction method is used to extract features, and training is performed based on the SVM classifier; the fusion feature extraction method includes extracting the edge gradient feature vector of the local target in the image data using the directional gradient histogram feature extraction method, extracting the image texture feature vector using the local binary pattern feature extraction method, and serially fusing the edge gradient feature vector of the local target with the image texture feature vector; Based on the classification detection model, the detection data sample set is classified and detected to obtain abnormal images of goods in and out of the material channel; The abnormal data marking and processing module for goods in the material channel is used to mark the abnormal positions of goods in and out of the material channel according to the abnormal images of goods in and out of the material channel, generate an abnormal position distribution positioning map, and process the material channel abnormalities according to the abnormal position distribution positioning map; The material channel wear maintenance cycle prediction module is used to score the material channel profile data based on a preset material channel wear scoring formula, generate a material channel wear maintenance prediction model based on the scoring results, and use the material channel wear maintenance prediction model to perform regular maintenance on the material channel.

2. The machine vision-based vending machine material channel monitoring system according to claim 1, characterized in that: The machine vision monitoring device includes a laser camera and several groups of charge-coupled device cameras. The laser camera is used to collect images of the side walls and bottom of the material channel; the charge-coupled device camera includes a first camera group and a second camera group; the first camera group is set at the entrance of the vending machine material channel and is used to capture images of goods at the entrance of the material channel; the second camera group is set in the vending machine material channel and is used to capture images of the goods shipping process in the material channel.

3. The machine vision-based vending machine material channel monitoring system according to claim 2, characterized in that: The material channel product data acquisition module also includes an image noise reduction processing evaluation unit and an abnormality type judgment unit; The image noise reduction evaluation unit is used to evaluate the noise reduction effect of the goods inlet and outlet channel images after noise reduction processing, and to screen the images and determine the abnormality type based on the evaluation results; including: Based on the preset subjective evaluation method and subjective evaluation rules, the image effect is evaluated to obtain a subjective evaluation score; The subjective evaluation method is for people to view the image; the subjective evaluation rules are based on the image noise attenuation status and whether the image edges are blurred; Based on a preset objective evaluation method and objective evaluation indicators, the image effect is evaluated to obtain an objective evaluation score; the objective evaluation method is to calculate the signal-to-noise ratio of the image, that is, to calculate the variance ratio of the signal to the noise; The objective evaluation index is a preset objective evaluation index formulated based on the change of image variance and the change of image signal-to-noise ratio value; The subjective evaluation score and the objective evaluation score are weighted and summed up to obtain the comprehensive evaluation score; If the comprehensive evaluation score is greater than or equal to the preset comprehensive evaluation score threshold, the image corresponding to the comprehensive evaluation score is retained; if the comprehensive evaluation score is less than the preset comprehensive evaluation score threshold, the image corresponding to the comprehensive evaluation score is discarded; Abnormal type judgment unit, used to judge the abnormality of out-of-stock and jammed goods in the material channel; After performing noise reduction on the product image at the feed channel, if the subjective evaluation score reaches the preset maximum value, it is determined that the feed channel is out of stock abnormally; If the subjective evaluation score is less than the preset maximum value, it is determined that the material channel is abnormally jammed.

4. The machine vision-based vending machine material channel monitoring system according to claim 2, characterized in that: The abnormal data acquisition module for goods in the material channel also includes a profile data acquisition unit and a profile data processing unit; A profile data acquisition unit, used for collecting profile data of the side wall and bottom images of the material channel using a profilometer; The profile data processing unit is used to process abnormal data points that do not meet the rules in a preset manner to obtain the material channel profile data; the preset method is: first delete the abnormal points, then use the data points before and after, and take the average for interpolation according to the rule that the data curve shows an overall downward trend.

5. The machine vision-based vending machine material channel monitoring system according to claim 1, characterized in that: The abnormal data marking and processing module of the material channel goods includes: an image marking classification unit, an image marking operation unit and a material channel abnormality processing unit; An image marking and classification unit is used to classify abnormal images of the goods inlet and outlet channels into two categories based on the clarity of the abnormal positions in the abnormal images of the goods inlet and outlet channels; images with a clarity greater than or equal to a preset clarity threshold are classified as clear images, and images with a clarity less than the preset clarity threshold are classified as blurred images; An image marking operation unit is used to mark the specific contours of the abnormality occurrence location in the clear image using a mask marking method, and generate a PNG format image file containing the mask trajectory information of the abnormality occurrence location; mark the detection area of the abnormality occurrence location in the blurred image using a detection frame marking method, and generate a JSON format text file containing the coordinates of the detection frame vertices and the abnormality location information; and aggregate the PNG format image file and the JSON format text file to generate an abnormality location distribution map; The material channel abnormality processing unit is used to process abnormal situations in the abnormal position distribution positioning map based on a preset processing solution.

6. The machine vision-based vending machine material channel monitoring system according to claim 1, characterized in that: The material channel wear maintenance cycle prediction module includes a scoring result generation unit, a cycle prediction model generation unit and a cycle prediction implementation unit; A scoring result generating unit is used to calculate the index parameter values for material channel damage scoring based on the profile data; the index parameter values include normal value, wear area, vertical wear value and side wear value; the index parameters are substituted into a preset scoring formula to obtain a scoring result; The cycle prediction model generation unit is used to calculate the monthly average of the scoring data within a cycle based on the scoring results, construct a time series, and then use the least squares method to fit it to generate a cycle prediction model; The cycle prediction implementation unit is used to predict the maintenance cycle according to the cycle prediction model to obtain the maintenance cycle.

7. The machine vision-based vending machine material channel monitoring system according to claim 1, characterized in that: It also includes a remote automatic processing module for automatically handling abnormal situations of some stuck goods; the remote automatic processing module includes a stuck goods risk determination unit, an abnormality warning processing unit and a material channel automatic adjustment unit; The cargo jam risk determination unit is used to set a corresponding cargo jam risk value based on the historical abnormal images of the cargo inlet and outlet channels, and to set a matching table between abnormal images and cargo jam risk values; to query the matching table to determine the cargo jam risk value corresponding to the current abnormal image of the cargo inlet and outlet channels; Obtain the total number of stuck cargo risk values, sum the total number of stuck cargo risk values, and obtain the cumulative stuck cargo risk value; Determine the cargo jam risk level based on the accumulated cargo jam risk value; The risk levels of stuck goods include high risk, medium risk and low risk; The early warning processing command sending unit is used to generate and send early warning processing commands to the material channel automatic adjustment unit based on the medium risk and low risk levels. The automatic channel adjustment unit is used to automatically handle the situation of goods jamming in the channel. The automatic channel adjustment unit includes a channel width fine-tuning device. When receiving a control command, the channel width fine-tuning device adjusts the channel width. When receiving a first control command corresponding to a low-risk warning, the channel width fine-tuning device adjusts the channel width according to a preset first adjustment value. When the second control command corresponding to the medium risk warning is received, the material channel width fine-tuning device adjusts the material channel width according to the preset second adjustment value.

8. The machine vision-based vending machine material channel monitoring system according to claim 1, characterized in that: It also includes a material channel maintenance effect detection unit for detecting the effect of material channel maintenance; The material channel maintenance effect detection unit includes a pre-maintenance test unit, a cleaning and maintenance unit, and a post-maintenance detection unit; The pre-maintenance test unit is used to inject a test fluid into the material channel and calculate the total resistance of the test fluid flowing in the material channel before the test based on the flow resistance and shape resistance of the test fluid; The cleaning and maintenance unit is used to clean and maintain the inside of the material channel by sweeping and dusting, fine grinding and leveling, dust removal and air drying and spraying; The post-maintenance detection unit is used to inject a test fluid into the material channel after cleaning and maintenance, and calculate the total resistance of the test fluid flowing in the material channel after maintenance by measuring the flow resistance and shape resistance of the test fluid. Obtaining the initial total resistance of the test fluid flowing in the material channel when the material channel is unused after leaving the factory, calculating the difference between the total resistance before the test and the initial total resistance to obtain a first difference; Calculating the difference between the total resistance after maintenance and the initial total resistance to obtain a second difference; Calculating the difference between the first difference and the second difference to obtain a third difference; When the third difference is less than or equal to the preset difference threshold, it is determined that the maintenance effect is qualified; otherwise, the cleaning and maintenance work is continued until the maintenance effect is determined to be qualified.

Images