A method and system for detecting display interface abnormalities of a vending machine

Abstract

The application discloses a kind of method and system for the detection of vending machine display interface exception, and monitoring camera device is arranged for the regional security monitoring of vending machine for various vending machines, although the video data obtained has obtained the vending interface information of vending machine, but this information is not specifically utilized.The application utilizes the video data of vending machine obtained by the monitoring camera to judge the working state of vending machine display interface, and detects whether vending display device works normally by recognizing and processing monitoring camera image.

Description

Technical Field

[0001] This invention relates to the field of vending machine maintenance technology, and in particular to a method and system for detecting abnormalities in the display interface of vending machines. Background Technology

[0002] In current technology, with the continuous upgrading and updating of vending machine technology, more and more vending machines have realized cashless payment functions and connected to the Internet for remote control. With the realization of these functions, the most obvious feature of the new generation of vending machines compared to older models is the use of more touch-screen displays, and a heavy reliance on the operation of these displays for purchasing goods. However, since vending machines are unattended devices, a problem arises: when these displays malfunction, it generally takes a long time to detect. For example, currently, display abnormalities are usually only discovered by operators during restocking operations or routine inspections. The sales losses incurred from the time the display malfunctions to the time operators repair and restore it to normal are substantial.

[0003] Often, sudden display anomalies are not caused by hardware failure. External electromagnetic interference such as lightning, high-power motors, power fluctuations, and electromagnetic radiation, or extreme weather conditions such as high temperatures, low temperatures, and high humidity, can easily cause sudden display anomalies that cannot be automatically recovered. Furthermore, the state of a malfunctioning display device is varied; sometimes it may result in a black screen, a distorted screen, abnormal stripes, or missing colors, or the display may freeze. Therefore, there is a need in the existing technology for a detection technique that can promptly and effectively detect various types of malfunctions in vending machine display devices, thereby improving the management efficiency of vending machine display devices. Summary of the Invention

[0004] One of the technical problems to be solved by this invention is to develop a detection technology that can detect various types of faults in the display devices of vending machines in a timely and effective manner.

[0005] To address the aforementioned technical problems, this invention provides a method for detecting abnormalities in the display interface of an automatic vending machine, the detection method comprising:

[0006] Step S100: Set positioning identification marks at the four top corners of the vending machine display device;

[0007] Step S101: Based on the image obtained by the monitoring camera of the vending machine, the image is extracted using the positioning identification mark to obtain the display interface images of multiple vending machine display devices;

[0008] Step S102: Perform image correction on the display interface images of the identified multiple vending machine display devices, and correct the tilted display device interface images into non-deformable images;

[0009] Step S103: Perform image pattern recognition on the corrected display interface image and determine whether the display interface status is black screen, distorted screen, abnormal stripes, missing colors, or normal display. If the determination result is black screen, distorted screen, abnormal stripes, or missing colors, the vending machine remote control server will notify the operation and maintenance personnel to conduct on-site inspection of the vending machine.

[0010] Step S104: If the judgment result of step S103 is that the display is normal, then the control device of the vending machine is instructed to take a screenshot of the display interface of the display device and upload the screenshot to the remote control server of the vending machine; at the same time, the monitoring video image at time t1 of the screenshot operation is extracted and the monitoring video image is extracted and corrected.

[0011] Step S105: Compare the screenshot at time t1 with the display interface extracted and corrected from the surveillance video image at time t1 to determine whether the two are the same display interface.

[0012] Step S106: If the judgment result in step S105 is a different display interface, it indicates that the vending machine display device has a screen freeze fault. In this case, the vending machine remote server controls the vending machine to restart the display device.

[0013] Step S107: After restarting in step S106, steps S104 and S105 are executed again. If the result is the same display interface, it means that the display device has returned to normal. If the result is still different, step S106 is executed again to restart the vending machine's display device and the abnormal count of the display device is increased by one.

[0014] Step S108: When the abnormal count of the display device exceeds the preset threshold, the vending machine remote control server notifies the operation and maintenance personnel to conduct on-site inspection of the vending machine.

[0015] In one embodiment, the image pattern recognition in step S103 employs a CNN-based artificial intelligence image recognition method.

[0016] According to another aspect of the present invention, a detection system for abnormal display interface of vending machine is also provided, the system including a processor and a memory, the memory storing program code for performing the following steps;

[0017] Step S100: Set positioning identification marks at the four top corners of the vending machine display device;

[0018] Step S101: Based on the image obtained by the monitoring camera of the vending machine, the image is extracted using the positioning identification mark to obtain the display interface images of multiple vending machine display devices;

[0019] Step S102: Perform image correction on the display interface images of the identified multiple vending machine display devices, and correct the tilted display device interface images into non-deformable images;

[0020] Step S103: Perform image pattern recognition on the corrected display interface image and determine whether the display interface status is black screen, distorted screen, abnormal stripes, missing colors, or normal display. If the determination result is black screen, distorted screen, abnormal stripes, or missing colors, the vending machine remote control server will notify the operation and maintenance personnel to conduct on-site inspection of the vending machine.

[0021] Step S104: If the judgment result of step S103 is that the display is normal, then the control device of the vending machine is instructed to take a screenshot of the display interface of the display device and upload the screenshot to the remote control server of the vending machine; at the same time, the monitoring video image at time t1 of the screenshot operation is extracted and the monitoring video image is extracted and corrected.

[0022] Step S105: Compare the screenshot at time t1 with the display interface extracted and corrected from the surveillance video image at time t1 to determine whether the two are the same display interface.

[0023] Step S106: If the judgment result in step S105 is a different display interface, it indicates that the vending machine display device has a screen freeze fault. In this case, the vending machine remote server controls the vending machine to restart the display device.

[0024] Step S107: After restarting in step S106, steps S104 and S105 are executed again. If the result is the same display interface, it means that the display device has returned to normal. If the result is still different, step S106 is executed again to restart the vending machine's display device and the abnormal count of the display device is increased by one.

[0025] Step S108: When the abnormal count of the display device exceeds the preset threshold, the vending machine remote control server notifies the operation and maintenance personnel to conduct on-site inspection of the vending machine.

[0026] Compared with the prior art, one or more embodiments of the present invention may have the following advantages:

[0027] 1. Surveillance cameras can be used for both normal equipment monitoring and image analysis to check the functionality of the vending machine's display. Technically, this is easy to implement, and equipment modification is relatively simple.

[0028] 2. By integrating big data learning and a control center, equipment malfunctions can be automatically identified and addressed. If automatic handling by the control center proves ineffective, maintenance personnel are then notified to handle the issue on-site, significantly reducing their inspection time and workload. This saves labor costs and improves work efficiency.

[0029] 3. Image detection methods have low computational requirements for image processing and low hardware requirements.

[0030] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0031] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0032] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;

[0033] Figure 2 This is a schematic diagram illustrating the setting of a location identification mark according to an embodiment of the present invention;

[0034] Figure 3 This is a schematic diagram of display interface image extraction and correction according to an embodiment of the present invention;

[0035] Figure 4 This is a schematic diagram comparing a screenshot of the display interface with a monitoring image according to an embodiment of the present invention;

[0036] Figure 5 This is a schematic diagram illustrating a method for handling a frozen display interface according to an embodiment of the present invention;

[0037] Figure 6 This is a schematic diagram illustrating the restart of the display after the display interface freezes according to an embodiment of the present invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the following description is provided in conjunction with the appendix. Figure 1-6 The present invention will be described in further detail below.

[0039] In existing technologies, various vending machines are equipped with surveillance cameras for regional security monitoring. Although the acquired video data contains information about the vending machine's interface, this information is not specifically utilized. In the embodiments of this invention, the video data acquired by the surveillance camera is used to determine the operational status of the vending machine's display interface. Figure 1 This is a flowchart of a method according to an embodiment of the present invention. The following is in conjunction with... Figure 1 This method will be explained.

[0040] First, step S100, as follows Figure 2 As shown, positioning identification markers are set at the four top corners of the vending machine display device. Due to the randomness of the installation position and angle of the surveillance camera, the vending machine's installation position, and the size and position of the display screen, the images captured by the surveillance camera have significant uncertainties. This undoubtedly increases the difficulty of image recognition and processing, affecting the implementation and accuracy of this embodiment. To solve this problem, positioning markers are installed at the four corners of the display to identify the effective screen area in the images captured by the surveillance camera. Furthermore, during image processing, the center coordinates of the four positioning marker points, i.e., the coordinates of the center pixel points, can be used to calculate the correction parameters for trapezoidal correction and image correction of the corresponding screen image.

[0041] Step S101, as follows Figure 3 As shown, based on the images acquired by the monitoring camera, image extraction is performed on the images using positioning and identification tags to obtain display interface images of multiple vending machine display devices.

[0042] Step S102: Perform image correction on the display interface images of the identified multiple vending machine display devices, and correct the tilted display device interface images into non-deformable images.

[0043] Step S103: Perform image pattern recognition on the corrected display interface image and determine whether the display interface status is black screen, distorted screen, abnormal stripes, missing colors, or normal display. If the determination result is black screen, distorted screen, abnormal stripes, or missing colors, the vending machine remote control server will notify the operation and maintenance personnel to conduct on-site inspection of the vending machine.

[0044] If the judgment result is a black screen, a distorted screen, abnormal stripes, or missing colors, it can be directly determined that the display is malfunctioning. However, the above pattern recognition process cannot directly determine the state of a frozen display interface. When the vending machine display interface freezes, the result of the above image pattern recognition will necessarily be that the display is normal. Therefore, this embodiment requires further judgment to determine whether the display device is in a frozen state.

[0045] Step S104, as follows Figure 4 As shown, if the judgment result of step S103 is that the display is normal, the control device of the vending machine is instructed to take a screenshot of the display interface of the display device and upload the screenshot to the remote control server of the vending machine; at the same time, the monitoring video image at time t1 of the screenshot operation is extracted, and the monitoring video image is extracted and corrected.

[0046] Step S105: Compare the screenshot at time t1 with the display interface extracted and corrected from the surveillance video image at time t1 to determine whether the two are the same display interface.

[0047] Step S106, as follows Figure 5 As shown, if the judgment result in step S105 is a different display interface, it indicates that the vending machine display device has a screen freeze fault, and the vending machine remote server controls the vending machine to restart the display device.

[0048] Step S107: After restarting in step S106, steps S104 and S105 are executed again. If the judgment result is the same display interface, it means that the display device has returned to normal. If the judgment result is still different, step S106 is executed again to restart the display device of the vending machine and the abnormal count of the display device is increased by one.

[0049] Step S108: When the abnormal count of the display device exceeds the preset threshold, the vending machine remote control server notifies the operation and maintenance personnel to conduct on-site inspection of the vending machine.

[0050] Therefore, the detection method for the vending machine display device in this embodiment can greatly reduce the failure maintenance time of the vending machine display device, and the remote control restart of the display device can reduce the workload of operators to carry out on-site maintenance.

[0051] The above description is merely a specific implementation example of the present invention, and the scope of protection of the present invention is not limited thereto. Any modifications or substitutions made to the present invention by those skilled in the art within the technical specifications described herein should be within the scope of protection of the present invention.

Claims

1. A method for detecting abnormalities in the display interface of an automatic vending machine, characterized in that, The detection method includes: Step S100: Set positioning identification marks at the four top corners of the vending machine display device; Step S101: Based on the image obtained by the monitoring camera of the vending machine, the image is extracted using the positioning identification mark to obtain the display interface images of multiple vending machine display devices; Step S102: Perform image correction on the display interface images of the identified multiple vending machine display devices, and correct the tilted display device interface images into non-deformable images; Step S103: Perform image pattern recognition on the corrected display interface image and determine whether the display interface status is black screen, distorted screen, abnormal stripes, missing color display, or normal display. If the determination result is black screen, distorted screen, abnormal stripes, or missing color display, the vending machine remote control server will notify the operation and maintenance personnel to conduct on-site inspection of the vending machine. Step S104: If the judgment result of step S103 is that the display is normal, then the control device of the vending machine is instructed to take a screenshot of the display interface of the display device and upload the screenshot to the remote control server of the vending machine; at the same time, the monitoring video image at time t1 of the screenshot operation is extracted and the monitoring video image is extracted and corrected. Step S105: Compare the screenshot at time t1 with the display interface extracted and corrected from the surveillance video image at time t1 to determine whether the two are the same display interface. Step S106: If the judgment result in step S105 is a different display interface, it indicates that the vending machine display device has a screen freeze fault. In this case, the vending machine remote server controls the vending machine to restart the display device. Step S107: After restarting in step S106, steps S104 and S105 are executed again. If the result is the same display interface, it means that the display device has returned to normal. If the result is still different, step S106 is executed again to restart the vending machine's display device and the abnormal count of the display device is increased by one. Step S108: When the abnormal count of the display device exceeds the preset threshold, the vending machine remote control server notifies the operation and maintenance personnel to conduct on-site inspection of the vending machine.

2. The method for detecting abnormalities in the display interface of an automatic vending machine according to claim 1, characterized in that, The image pattern recognition in step S103 employs a CNN-based artificial intelligence image recognition method.

3. A detection system for abnormal display interface of vending machine, the system comprising a processor and a memory, the memory storing program code for performing the following steps; Step S100: Set positioning identification marks at the four top corners of the vending machine display device; Step S101: Based on the image obtained by the monitoring camera of the vending machine, the image is extracted using the positioning identification mark to obtain the display interface images of multiple vending machine display devices; Step S102: Perform image correction on the display interface images of the identified multiple vending machine display devices, and correct the tilted display device interface images into non-deformable images; Step S103: Perform image pattern recognition on the corrected display interface image and determine whether the display interface status is black screen, distorted screen, abnormal stripes, missing color display, or normal display. If the determination result is black screen, distorted screen, abnormal stripes, or missing color display, the vending machine remote control server will notify the operation and maintenance personnel to conduct on-site inspection of the vending machine. Step S104: If the judgment result of step S103 is that the display is normal, then the control device of the vending machine is instructed to take a screenshot of the display interface of the display device and upload the screenshot to the remote control server of the vending machine; at the same time, the monitoring video image at time t1 of the screenshot operation is extracted and the monitoring video image is extracted and corrected. Step S105: Compare the screenshot at time t1 with the display interface extracted and corrected from the surveillance video image at time t1 to determine whether the two are the same display interface. Step S106: If the judgment result in step S105 is a different display interface, it indicates that the vending machine display device has a screen freeze fault. In this case, the vending machine remote server controls the vending machine to restart the display device. Step S107: After restarting in step S106, steps S104 and S105 are executed again. If the result is the same display interface, it means that the display device has returned to normal. If the result is still different, step S106 is executed again to restart the vending machine's display device and the abnormal count of the display device is increased by one. Step S108: When the abnormal count of the display device exceeds the preset threshold, the vending machine remote control server notifies the operation and maintenance personnel to conduct on-site inspection of the vending machine.

Images