Universal method for collecting data of display screen of oiling machine based on computer vision
Through the computer vision-based dual YOLO model and MQTT protocol, the universality and real-time issues of digital detection on the liquid crystal screen of the fuel dispenser are solved, and efficient and accurate data collection and transmission are achieved.
Patent Information
- Application Number
- CN202510928132.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-07
- Publication Date
- 2025-09-26
AI Technical Summary
The existing fuel dispenser hub system lacks versatility, has low communication rates, large data transmission delays, high implementation and maintenance costs, poor LCD screen digital detection in complex environments, inefficient dynamic digital update detection, and insufficient real-time data transmission.
A computer vision-based method is adopted, using dual YOLO models to detect LCD screen areas and recognize numbers. The digital sorting algorithm and MQTT protocol are combined to achieve real-time data transmission, ensuring recognition accuracy and real-time performance.
It achieves efficient and accurate collection and verification of LCD screen numbers in complex environments, improves recognition accuracy and efficiency, ensures data accuracy and real-time performance, and reduces the possibility of misidentification.
Smart Images
Figure CN120708231A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of gas station data management, and in particular to a general method for collecting data from a gas station display screen based on computer vision. Background Art
[0002] Currently, fueling data collection at gas stations typically relies on a fuel dispenser hub. This hardware device, installed at each fuel dispenser, aggregates and transmits fueling information. It collects key data such as fuel volume, payment amount, and time, and transmits this information to the gas station management system. The hub interacts with a central server or management terminal via standard or proprietary communication protocols, enabling remote monitoring and management of gas station data.
[0003] This solution has achieved, to a certain extent, the automated collection and management of gas station refueling data. Especially in larger gas stations, the hub system can greatly reduce manual intervention and improve the efficiency and accuracy of data collection.
[0004] Common communication protocols for hubs mainly include two parts: Proprietary protocols: Different brands of fuel dispensers often use manufacturer-defined communication protocols suitable for specific models of equipment. Standardized protocols: Some fuel dispensers support standardized industrial communication protocols such as Modbus and CAN bus for communication between devices.
[0005] The existing technology has the following defects or problems:
[0006] Existing fuel dispenser hub technology has many shortcomings. First, due to the differences in equipment brands and communication protocols at different gas stations, the system lacks universality, making standardized integration difficult. Second, the hub's communication rate is low, and data transmission delays are large, which affects the effectiveness of real-time monitoring. In addition, implementation and maintenance costs are high, especially when adapting to multiple devices, making system upgrades and maintenance complex and expensive. In addition, existing LCD screen digital detection technology is generally based on image processing and OCR (optical character recognition). However, in complex environments, such as gas stations, factors such as lighting changes, screen reflections, weather conditions, and camera angles can interfere with detection results. Furthermore, existing technical solutions are not efficient enough for detecting dynamic digital updates on LCD screens, and the real-time nature of data transmission is not fully addressed.
[0007] It should be noted that the above content falls within the technical knowledge of the inventor and does not necessarily constitute prior art. Summary of the Invention
[0008] In response to the shortcomings of the existing technology, the present invention provides a general method for collecting fuel dispenser display screen data based on computer vision, which solves the current problems.
[0009] To achieve the above objectives, the present invention provides the following technical solution: a general method for collecting fuel dispenser display screen data based on computer vision, comprising the following steps:
[0010] Step 1: Use a camera to capture the real-time video stream of the gas station LCD screen as input for subsequent digital detection;
[0011] Step 2: Use the first set of YOLO models to detect the area of the LCD screen in the video frame and identify the position and boundaries of the screen;
[0012] Step 3: After extracting the LCD screen area, use the second set of YOLO models to recognize the numbers on the LCD screen;
[0013] Step 4: Process the identified numbers and use grouping and sorting strategies to calculate whether the refueling quantity, unit price and total amount are consistent;
[0014] Step 5: The processed refueling data is transmitted to the server in real time via the MQTT protocol for storage and further analysis.
[0015] In some embodiments, the LCD screen at the gas station is typically used to display key refueling data, including the amount of refueling, unit price, and total amount. The video stream is used as input to the YOLO model to identify and extract data information. The video data is collected through a continuous connection between the camera and the processing system to ensure real-time update and timely transmission of the data.
[0016] In some embodiments, the YOLO model, as a deep learning-based object detection technology, detects and classifies multiple objects in an image through a single feedforward neural network operation and generates corresponding bounding boxes;
[0017] The specific steps are as follows:
[0018] 1) After a video frame is input into the system, the first set of YOLO models is used to detect the LCD screen in the image. The system passes each video frame to the YOLO model, which uses the feature extraction mechanism of the convolutional neural network to perform a global analysis of the image and generate a series of possible candidate regions;
[0019] 2) The YOLO model outputs a bounding box for each region, representing the detected target location. The system will focus on filtering out the bounding box where the LCD screen is located. To improve recognition accuracy, the system sets a confidence threshold. Only bounding boxes with a confidence level exceeding the threshold are retained.
[0020] In some embodiments, the extracted LCD screen image is passed as input to a second set of YOLO models, which are specially trained to recognize digital characters 0-9;
[0021] The specific steps for identification are as follows:
[0022] 1) The YOLO model generates multiple predicted bounding boxes through global feature analysis. Each box represents a detected digit and is accompanied by a category label, i.e., a specific digit value.
[0023] 2) The system applies the confidence threshold again to filter out bounding boxes with low confidence, ensuring that only high-confidence digit recognition results are retained;
[0024] 3) The system sorts the numbers according to their X coordinates, ensuring that the numbers in the same row are arranged in order from left to right to correctly reconstruct the number string displayed on the LCD screen.
[0025] In some embodiments, after the LCD screen digital recognition is completed, the system will process and verify the detected digital data to ensure that the recognition result is consistent with the actual displayed information, and perform subsequent logical calculations. The specific steps are as follows:
[0026] 1) The system groups the identified number boxes according to their coordinates on the screen. It analyzes the Y coordinate of each number and groups the numbers in the same row into one group.
[0027] 2) The system sorts the number boxes in each row from left to right according to the X coordinate, ensuring that the numbers in the same row are arranged in the correct order, so that the complete digital information displayed on the LCD can be reconstructed;
[0028] 3) After grouping and sorting the numbers, the numbers are processed and the identified numbers are categorized into the corresponding fields for typical display content on a gas station LCD screen, including fuel quantity, unit price, and total amount;
[0029] Refueling quantity and unit price processing: After the identified numbers are identified and classified, the refueling quantity and unit price are multiplied and a theoretical total amount is calculated through basic arithmetic operations;
[0030] Total amount verification: Compare the theoretical total amount obtained by the above calculation with the total amount displayed on the LCD screen. If the calculation result is consistent with the recognized total amount, it proves that the recognized refueling quantity, unit price and total amount are correct. Otherwise, it will be judged that there is an error in the recognition.
[0031] 4) This process is repeated during the processing of each frame to ensure that the digital data in each frame has been processed and verified. This operation is performed frame by frame, and the digital recognition results of each frame are processed independently and logically checked. If a recognition error occurs in a frame, further adjustments and corrections can be made by comparing the results of the previous and next frames.
[0032] 5) The processed and verified data is prepared for subsequent steps, including data storage and transmission, ensuring that the digital information displayed on the gas station LCD screen is accurately transmitted to the server and monitoring terminal.
[0033] In some embodiments, the specific steps are as follows:
[0034] 1) Using an already connected MQTT client, encapsulate the identified fuel quantity, unit price, and total amount data in lightweight JSON format and publish it to a predefined MQTT body;
[0035] Specifically, create and configure an MQTT client instance, connect to the specified MQTT server by setting a unique client ID, and set relevant connection parameters, including the broker address, port number, protocol version, and connection duration. After the client successfully connects, confirm the connection status through the on_connect callback function, subscribe to the specified MQTT topic, and wait for receiving and sending data.
[0036] 2) Once the data is ready and verified, the publish function is called to send the message. This function encapsulates the processed data into a JSON object, which includes the amount of gas, unit price, and total amount. This data is mapped to a predefined MQTT topic. If the data is sent successfully, a success message will be printed in the console. Otherwise, an error message will be displayed and a reconnection attempt will be made.
[0037] 3) During the message sending process, ensure that the MQTT client always maintains a connection with the Broker;
[0038] When an interruption occurs, the on_disconnect callback function will attempt to automatically reconnect, setting an increasing reconnection delay and retrying up to 12 times until the connection is reestablished or the operation is abandoned. This mechanism ensures that even in unstable network conditions, the gas station's LCD screen data can still be sent stably, ensuring the real-time and reliability of the data.
[0039] 4) Refueling data transmitted via the MQTT protocol will be sent to a designated server in real time for further processing, storage, or display, ensuring that important refueling information displayed on the LCD screen can be effectively delivered.
[0040] Compared with the existing technology, the present invention provides a general method for collecting data from fuel dispenser displays based on computer vision, which has the following beneficial effects:
[0041] This general method for collecting data from gas station display screens based on computer vision adopts a dual YOLO model architecture and a digital verification mechanism to achieve efficient recognition and verification of numbers on gas station LCD screens. First, through the first YOLO model, the system can accurately detect the LCD screen area in the video stream, filter out background interference and extract the screen content. Next, the second YOLO model is specifically used to identify digital information on the LCD screen, ensuring that digital content can still be accurately extracted in various complex environments, such as lighting changes, screen reflections, etc. To ensure the correctness of the numbers, the system introduces a digital sorting algorithm to sort the numbers according to the X-coordinate on the screen, ensuring that the numbers in the same row are arranged in order from left to right to avoid misidentification. Afterwards, the system performs real-time verification of the identified refueling quantity, unit price and total amount, uses multiplication operations to calculate the refueling quantity and unit price, and compares the calculation results with the total amount displayed on the LCD screen to ensure the accuracy of the recognized data;
[0042] The application of a dual-model architecture and digital verification mechanism enables accurate collection and verification of numbers on gas station LCD screens in complex environments. First, the use of dual models significantly improves recognition accuracy and efficiency. The first model precisely detects the screen area, while the second model specifically recognizes digital information, thus resolving interference from external factors such as lighting changes and reflections. Second, the digital sorting algorithm ensures the correct arrangement of numbers on the screen, avoiding recognition errors caused by misplaced numbers.
[0043] Furthermore, real-time verification of fuel quantity, unit price, and total amount ensures data accuracy. If the recognition result does not match expectations, the system automatically issues an alarm. Multi-frame information fusion further enhances recognition robustness, reducing the possibility of misidentification. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Figure 1 This is a basic flow chart of the general method for collecting display screen data according to the present invention;
[0045] Figure 2 This is a flow chart of digital data processing and verification of the present invention. DETAILED DESCRIPTION
[0046] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the embodiments of the present invention and the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.
[0047] It should be understood that the step numbers used herein are only for convenience of description and are not intended to limit the order in which the steps are executed.
[0048] It should be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise.
[0049] The terms “include” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.
[0050] The term "and / or" refers to and includes any and all possible combinations of one or more of the associated listed items.
[0051] See also Figure 1-2 In this embodiment, a general method for collecting data from a fuel dispenser display screen based on computer vision includes the following steps:
[0052] Step 1: Use a camera to capture the real-time video stream of the gas station LCD screen as input for subsequent digital detection;
[0053] Gas station LCD screens typically display key refueling data, including fuel quantity, unit price, and total amount. Video streams are used as input for the YOLO model to identify and extract data information. Video data is collected through a continuous connection between the camera and the processing system, ensuring real-time data updates and timely delivery.
[0054] Step 2: Use the first set of YOLO models to detect the area of the LCD screen in the video frame and identify the position and boundaries of the screen;
[0055] As a target detection technology based on deep learning, the YOLO model detects and classifies multiple targets in an image through a single feedforward neural network operation and generates corresponding bounding boxes.
[0056] The specific steps are as follows:
[0057] 1) After a video frame is input into the system, the first set of YOLO models is used to detect the LCD screen in the image. The system passes each video frame to the YOLO model, which uses the feature extraction mechanism of the convolutional neural network to perform a global analysis of the image and generate a series of possible candidate regions;
[0058] 2) The YOLO model outputs a bounding box for each region, representing the detected target location. The system will focus on selecting the bounding box containing the LCD screen. To improve recognition accuracy, the system sets a confidence threshold, and only bounding boxes with a confidence level exceeding the threshold are retained.
[0059] In this way, the system can accurately extract the LCD screen area in the video frame. The extracted LCD screen area will serve as input for subsequent steps, providing a precise target image for the recognition of numbers on the screen, and avoiding other interference information that affects the accuracy of digital detection.
[0060] Step 3: After extracting the LCD screen area, use the second set of YOLO models to recognize the numbers on the LCD screen;
[0061] The extracted LCD screen image is passed as input to the second set of YOLO models, which are specially trained to recognize the digital characters 0-9;
[0062] The specific steps for identification are as follows:
[0063] 1) The YOLO model generates multiple predicted bounding boxes through global feature analysis. Each box represents a detected digit and is accompanied by a category label, i.e., a specific digit value.
[0064] 2) The system applies the confidence threshold again to filter out bounding boxes with low confidence, ensuring that only high-confidence digit recognition results are retained;
[0065] 3) The system sorts the numbers according to their X coordinates, ensuring that the numbers in the same row are arranged in order from left to right to correctly reconstruct the number string displayed on the LCD screen;
[0066] Through this method, the system can extract and reconstruct complete digital information from the LCD screen area, providing key data such as refueling volume, unit price and total amount for subsequent processing and verification.
[0067] Step 4: Process the identified numbers and use grouping and sorting strategies to calculate whether the refueling quantity, unit price and total amount are consistent;
[0068] After the LCD screen digital recognition is completed, the system will process and verify the detected digital data to ensure that the recognition result is consistent with the actual displayed information, and perform subsequent logical calculations. The specific steps are as follows:
[0069] 1) The system groups the identified number boxes according to their coordinates on the screen. It analyzes the Y coordinate of each number and groups the numbers in the same row into one group.
[0070] 2) The system sorts the number boxes in each row from left to right according to the X coordinate, ensuring that the numbers in the same row are arranged in the correct order, so that the complete digital information displayed on the LCD can be reconstructed;
[0071] 3) After grouping and sorting the numbers, the numbers are processed and the identified numbers are categorized into the corresponding fields for typical display content on a gas station LCD screen, including fuel quantity, unit price, and total amount;
[0072] Refueling quantity and unit price processing: After the identified numbers are identified and classified, the refueling quantity and unit price are multiplied and a theoretical total amount is calculated through basic arithmetic operations;
[0073] Total amount verification: Compare the theoretical total amount obtained by the above calculation with the total amount displayed on the LCD screen. If the calculation result is consistent with the recognized total amount, it proves that the recognized refueling quantity, unit price and total amount are correct. Otherwise, it will be judged that there is an error in the recognition.
[0074] 4) This process is repeated during the processing of each frame to ensure that the digital data in each frame has been processed and verified. This operation is performed frame by frame, and the digital recognition results of each frame are processed independently and logically checked. If a recognition error occurs in a frame, further adjustments and corrections can be made by comparing the results of the previous and next frames.
[0075] 5) The processed and verified data is prepared for subsequent steps, including data storage and transmission, ensuring that the digital information displayed on the gas station LCD screen is accurately transmitted to the server and monitoring terminal;
[0076] Step 5: The processed refueling data is transmitted to the server in real time via the MQTT protocol for storage and further analysis;
[0077] The specific steps are as follows:
[0078] 1) Using an already connected MQTT client, encapsulate the identified fuel quantity, unit price, and total amount data in lightweight JSON format and publish it to a predefined MQTT body;
[0079] Specifically, create and configure an MQTT client instance, connect to the specified MQTT server by setting a unique client ID, and set relevant connection parameters, including the broker address, port number, protocol version, and connection duration. After the client successfully connects, confirm the connection status through the on_connect callback function, subscribe to the specified MQTT topic, and wait for receiving and sending data.
[0080] 2) Once the data is ready and verified, the publish function is called to send the message. This function encapsulates the processed data into a JSON object, which includes the amount of gas, unit price, and total amount. This data is mapped to a predefined MQTT topic. If the data is sent successfully, a success message will be printed in the console. Otherwise, an error message will be displayed and a reconnection attempt will be made.
[0081] 3) During the message sending process, ensure that the MQTT client always maintains a connection with the Broker;
[0082] When an interruption occurs, the on_disconnect callback function will attempt to automatically reconnect, setting an increasing reconnection delay and retrying up to 12 times until the connection is reestablished or the operation is abandoned. This mechanism ensures that even in unstable network conditions, the gas station's LCD screen data can still be sent stably, ensuring the real-time and reliability of the data.
[0083] 4) Refueling data transmitted via the MQTT protocol will be sent to a designated server in real time for further processing, storage, or display, ensuring that important refueling information displayed on the LCD screen can be effectively delivered.
[0084] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are described briefly because they are generally similar to the method embodiments. For relevant parts, refer to the description of the method embodiments.
[0085] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art will be able to modify the technical solutions described in the aforementioned embodiments or substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.
Claims
1. A general method for collecting data from a fuel dispenser display screen based on computer vision, characterized in that: The steps include: Step 1: Use a camera to capture the real-time video stream of the gas station LCD screen as input for subsequent digital detection; Step 2: Use the first set of YOLO models to detect the area of the LCD screen in the video frame and identify the position and boundaries of the screen; Step 3: After extracting the LCD screen area, use the second set of YOLO models to recognize the numbers on the LCD screen; Step 4: Process the identified numbers and use grouping and sorting strategies to calculate whether the refueling quantity, unit price and total amount are consistent; Step 5: The processed refueling data is transmitted to the server in real time via the MQTT protocol for storage and further analysis.
2. The general method for collecting data from a fuel dispenser display screen based on computer vision according to claim 1 is characterized in that: The LCD screen at the gas station described in step 1 is usually used to display key refueling data, including the amount of fuel, unit price, and total amount. The video stream is used as input to the YOLO model to identify and extract data information. The video data is collected through a continuous connection between the camera and the processing system to ensure real-time data updates and timely transmission.
3. The general method for collecting data from a fuel dispenser display screen based on computer vision according to claim 1 is characterized in that: The YOLO model described in step 2 is a deep learning-based target detection technology that detects and classifies multiple targets in an image through a single feedforward neural network operation and generates corresponding bounding boxes. The specific steps are as follows: 1) After a video frame is input into the system, the first set of YOLO models is used to detect the LCD screen in the image. The system passes each video frame to the YOLO model, which uses the feature extraction mechanism of the convolutional neural network to perform a global analysis of the image and generate a series of possible candidate regions; 2) The YOLO model outputs a bounding box for each region, representing the detected target location. The system will focus on filtering out the bounding box where the LCD screen is located. To improve recognition accuracy, the system sets a confidence threshold. Only bounding boxes with a confidence level exceeding the threshold are retained.
4. The general method for collecting data from a fuel dispenser display screen based on computer vision according to claim 1 is characterized in that: The extracted LCD screen image described in step 3 is passed as input to the second set of YOLO models, which are specially trained to recognize the digits 0-9. The specific steps for identification are as follows: 1) The YOLO model generates multiple predicted bounding boxes through global feature analysis. Each box represents a detected digit and is accompanied by a category label, i.e., a specific digit value. 2) The system applies the confidence threshold again to filter out bounding boxes with low confidence, ensuring that only high-confidence digit recognition results are retained; 3) The system sorts the numbers according to their X coordinates, ensuring that the numbers in the same row are arranged in order from left to right to correctly reconstruct the number string displayed on the LCD screen.
5. The general method for collecting data from a fuel dispenser display screen based on computer vision according to claim 1 is characterized in that: After the LCD screen digital recognition is completed in step 4, the system will process and verify the detected digital data to ensure that the recognition result is consistent with the actual displayed information, and perform subsequent logical calculations. The specific steps are as follows: 1) The system groups the identified number boxes according to their coordinates on the screen. It analyzes the Y coordinate of each number and groups the numbers in the same row into one group. 2) The system sorts the number boxes in each row from left to right according to the X coordinate, ensuring that the numbers in the same row are arranged in the correct order, so that the complete digital information displayed on the LCD can be reconstructed; 3) After grouping and sorting the numbers, the numbers are processed and the identified numbers are categorized into the corresponding fields for typical display content on a gas station LCD screen, including fuel quantity, unit price, and total amount; Refueling quantity and unit price processing: After the identified numbers are identified and classified, the refueling quantity and unit price are multiplied and a theoretical total amount is calculated through basic arithmetic operations; Total amount verification: Compare the theoretical total amount obtained by the above calculation with the total amount displayed on the LCD screen. If the calculation result is consistent with the recognized total amount, it proves that the recognized refueling quantity, unit price and total amount are correct. Otherwise, it will be judged that there is an error in the recognition. 4) This process is repeated during the processing of each frame to ensure that the digital data in each frame has been processed and verified. This operation is performed frame by frame, and the digital recognition results of each frame are processed independently and logically checked. If a recognition error occurs in a frame, further adjustments and corrections can be made by comparing the results of the previous and next frames. 5) The processed and verified data is prepared for use in subsequent steps, including data storage and transmission, ensuring that the digital information displayed on the gas station LCD screen is accurately transmitted to the server and monitoring terminal.
6. The general method for collecting data from a fuel dispenser display screen based on computer vision according to claim 1 is characterized in that: The specific steps described in step 5 are as follows: 1) Using an already connected MQTT client, encapsulate the identified fuel quantity, unit price, and total amount data in lightweight JSON format and publish it to a predefined MQTT body; Specifically, create and configure an MQTT client instance, connect to the specified MQTT server by setting a unique client ID, and set relevant connection parameters, including the broker address, port number, protocol version, and connection duration. After the client successfully connects, confirm the connection status through the on_connect callback function, subscribe to the specified MQTT topic, and wait for receiving and sending data. 2) Once the data is ready and verified, the publish function is called to send the message. This function encapsulates the processed data into a JSON object, which includes the amount of gas, unit price, and total amount. This data is mapped to a predefined MQTT topic. If the data is sent successfully, a success message will be printed in the console. Otherwise, an error message will be displayed and a reconnection attempt will be made. 3) During the message sending process, ensure that the MQTT client always maintains a connection with the Broker; When an interruption occurs, the on_disconnect callback function will attempt to automatically reconnect, setting an increasing reconnection delay and retrying up to 12 times until the connection is reestablished or the operation is abandoned. This mechanism ensures that even in unstable network conditions, the gas station's LCD screen data can still be sent stably, ensuring the real-time and reliability of the data. 4) Refueling data transmitted via the MQTT protocol will be sent to a designated server in real time for further processing, storage, or display, ensuring that important refueling information displayed on the LCD screen can be effectively delivered.