A multi-link gas station cashing behavior identification system

Abstract

The present application relates to the field of information processing, and provides a multi-link gas station cashing behavior identification system.The system comprises four modules: a configuration module, an image analysis module, a matching module and an alarm module: the deep learning technology in computer vision is used to set standard position information in the gas station for cashing behavior detection, which reduces the investment of enterprise manual searching, and at the same time, the visual monitoring system is connected with the order system to obtain order information, which is accurately matched with the cashing behavior to determine whether it is a cashing behavior, and finally determines whether it is a cashing behavior, and realizes full-automatic accurate judgment.

Description

Technical Field

[0001] This invention relates to the field of information processing, and specifically to a multi-linked gas station cash-out behavior identification system. Background Technology

[0002] Addressing the issue of employees cashing out money presents practical challenges. Currently, the main method for confirming cash-out is to manually check gas station video recordings to find instances of gas station attendants receiving cash, and then search the order system for corresponding orders to determine if cash-out has occurred. However, cash-receiving behavior itself is short-lived, and searching through video recordings is time-consuming. When there are many monitoring devices, more manpower is required, resulting in significant investment and minimal returns.

[0003] Therefore, how to provide an AI-based system for detecting cash-out behavior is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] To address at least one of the aforementioned technical problems, this application proposes a multi-linked gas station cash-out behavior recognition system, comprising four modules: a configuration module, an image analysis module, a matching module, and an alarm module.

[0005] The configuration module acquires images from the gas station's on-site cameras and marks the refueling island area, the corresponding fuel nozzle number, and the ID information of the person picking up the nozzle in the image. There are at least two cameras.

[0006] The image analysis module uses a deep learning model to analyze the decoded video data in real time and identify cash receiving behavior.

[0007] The matching module obtains order information from the gas station's order system and information confirming the occurrence of cash collection from the image analysis module. It then matches the fuel nozzle number and cash collection time uploaded by the image analysis module with the fuel nozzle number and nozzle lifting / hanging time in the order information to obtain the order information corresponding to the cash collection behavior.

[0008] The alarm module judges the matched order information, and when it detects cash-out behavior, it transmits the alarm information to the gas station's management platform.

[0009] Preferably, the alarm information includes at least order information, the ID of the person who picked up the gun, and video image information of the person who picked up the gun.

[0010] Preferably, the configuration module acquires the on-site camera footage of the gas station at the front end and marks the refueling island area, the corresponding fuel nozzle number information, and the ID identification information of the person picking up the nozzle in the image. This includes: starting the front-end streaming media service, configuring the camera IP address, acquiring the on-site image using the streaming media RTSP protocol, marking the refueling island area based on the canvas API, and marking the fuel nozzle number information corresponding to the refueling island.

[0011] Preferably, the image analysis module's method of using a deep learning model to analyze and decode video data in real time to identify cash-receiving behavior includes: the on-site equipment decodes camera video data in real time through a chip, transmits the data to a neural network analysis model for analysis, and when a vehicle is detected entering the refueling area, the vehicle tracking algorithm tracks the vehicle, and after the vehicle stops at the refueling island, the cash-receiving behavior monitoring is activated.

[0012] Preferably, the monitoring of cash collection behavior includes: after a customer gets off the vehicle, the pedestrian tracking algorithm starts to track and identify the customer, and identifies whether there is cash being handed over on the ground.

[0013] Preferably, the ground cash receiving recognition identifies whether there is a cash receiving or passing behavior on the ground. Specifically, it uses a neural network analysis model to analyze and detect the hand behavior of the person carrying the gun and the customer, and obtain an image of the hand quadrilateral.

[0014] Preferably, the matching module obtains the order information corresponding to the cash payment behavior by: obtaining the refueling order information from the order system through the ROCKETMQ message queue, including the fuel nozzle number, lifting and grabbing information, payment method, and payment amount; and matching the cash payment time and fuel nozzle number information obtained through the image analysis module with the fuel nozzle number and lifting and grabbing time from the order information to obtain the refueling order information corresponding to the cash payment refueling process.

[0015] Preferably, a neural network analysis model is used to analyze and detect the hand behavior of the person picking up the gun and the customer, and to obtain the area image of the hand quadrilateral. The acquisition of the hand quadrilateral image information is based on the transformation of the logo image in the gas station and then compared with the standard image to determine whether it is a cash image.

[0016] Preferably, a deformation matrix is ​​preset based on the location information of the signage image set at the gas station and the angle information of the camera. When extracting the quadrilateral information image of the user's hand, if the preset deformation matrix exists, the deformation transformation is directly performed and then compared with the standard image; if the preset deformation matrix does not exist, the deformation matrix is ​​calculated based on the signage image and the signage image obtained from the camera.

[0017] Preferably, the neural network analysis module includes at least one convolutional layer, at least one fully connected layer, and a softmax layer; the feature map of the last convolutional layer is input into the at least one fully connected layer to obtain a two-dimensional vector output by the last fully connected layer in the at least one fully connected layer.

[0018] To address this, the present invention provides a multi-linked gas station cash-out behavior identification system. This application utilizes deep learning technology from computer vision for cash-receiving behavior detection. Standard location information and user identification information are set up within the gas station for cash-receiving behavior detection, reducing the need for manual searching. Simultaneously, the visual monitoring system is integrated with the order system to acquire order information. By accurately matching cash-receiving behavior with order information, a judgment is made to determine whether it constitutes cash-out behavior, achieving fully automated and accurate judgment. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the system structure.

[0021] Figure 2 This is the implementation flow of the image analysis module in this system. Implementation

[0022] Referring to the following description and accompanying drawings, these and other features and characteristics, methods of operation, functions of related elements of the structure, combinations of parts, and economics of manufacture of this disclosure can be better understood, wherein the description and drawings form part of the specification. However, it is clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this disclosure. It is understood that the drawings are not drawn to scale. Various structural diagrams are used in this disclosure to illustrate various variations of embodiments according to this disclosure.

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] It should be noted that the " / " in this article means "or". For example, A / B can mean A or B. The "and / or" in this article is only a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone.

[0025] It should be noted that, in order to clearly describe the technical solutions of the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish the same or similar items with essentially the same function or effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order. For example, first information and second information are used to distinguish different information, rather than to describe a specific order of information.

[0026] It should be noted that in the embodiments of the present invention, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present invention should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0027] like Figure 1 As shown, the present invention provides a multi-linkage gas station cash-out behavior recognition system, which includes four modules: configuration module, image analysis module, matching module, and alarm module.

[0028] Module 1, Configuration Module: This module acquires images from the gas station's on-site cameras at the front end and marks the refueling island area and the corresponding fuel nozzle number on the image.

[0029] Module 2, Image Analysis Module, utilizes a deep learning model to analyze decoded video data in real time to identify cash receiving behavior.

[0030] The deep learning model analyzes and decodes the video data in real time as follows:

[0031] Assuming there are P consecutive frames of images, automatically detected images containing preset feature images are used to continuously analyze the shape features of the hand and other objects, extracting the object features from these consecutive images. Since the video is continuous, the coordinates of the extracted hand features within each frame within the preset range will not be completely consistent as the video frame moves. Therefore, for each of the P frames automatically detected to contain suspected feature images, based on standard location information (such as at a gas station location) and camera shooting calibration information, the obtained image deformation information is transformed using stretching, shrinking, and perspective transformation to adjust the shape of the cash image to the optimal cash image shape, thereby performing a comparison with a standard cash image.

[0032] Preferably, the coordinates of four points in the extracted image can be adjusted to (x1, y1), (x2, y2), (x3, y3), and (x4, y4), and the area enclosed by these four points is compared with the optimal cash image. During the deformation process, standard image information of the gas station's set location is introduced. For example, the sign information is a reference image of the sign, and the deformation parameter matrix is ​​obtained based on the reference image of the sign, thereby obtaining a graphic similar to a linear image or a general black and white grid image that the user has at hand.

[0033] The matching detection of the feature images can employ a convolutional neural network algorithm. Specifically, the neural network algorithm involves: inputting the i-th frame image from the M frames into at least one convolutional layer of a first convolutional neural network model to obtain the feature map of the last convolutional layer. The first convolutional neural network model includes at least one convolutional layer, at least one fully connected layer, and one softmax layer. The feature map of the last convolutional layer is then input into the at least one fully connected layer to obtain a two-dimensional vector output by the last fully connected layer. Finally, the two-dimensional vector is input into the softmax layer to obtain a vector used to identify whether the i-th frame image contains the cash.

[0034] The weight parameters of the convolutional layers, the bias parameters of the convolutional layers, the weight parameters of the fully connected layers, and the bias parameters of the fully connected layers in the first convolutional neural network model are generated based on pre-set images containing the cash and images not containing the cash. During the training process, for video information taken from different angles of the entire gas station, the captured images are preset standard images within the gas station at a set location. The standard image can be a black and white grid in the video test. Based on the mapping between the feature image and the camera position and the shooting position, the transformation matrix, i.e., the deformation matrix, is determined in subsequent processing such as zooming in, by referring to the coordinate point information inside the gas station.

[0035] Module 3, the matching module, obtains order information from the order system and cash collection information from the analysis module. It matches the oil nozzle number and cash collection time uploaded by the analysis module with the oil nozzle number and oil nozzle lifting / hanging time in the order information to obtain the order information corresponding to the cash collection behavior.

[0036] Module 4, Alarm Module: The alarm module judges the matched order information. If cash-out behavior is detected, the alarm information is transmitted to the management platform.

[0037] Based on the above modules, this system can process video stream data from cameras in real time, monitor whether cash-for-refueling transactions occur during the refueling process, and then match the refueling order information for cash-for-refueling transactions to determine whether the corresponding order information is associated with cash-for-refueling behavior. If associated with cash-for-refueling behavior, an alert is issued. The alert information includes at least the order information, the ID of the person picking up the gun, and the video image information of the person picking up the gun.

[0038] The method for "acquiring on-site camera footage of the gas station from the front end, and marking the refueling island area and the corresponding fuel nozzle number information in the footage" described in Module 1 is as follows: Start the front-end streaming media service, configure the camera IP address, acquire on-site footage using the streaming media RTSP protocol, mark the refueling island area based on the canvas API, and mark the fuel nozzle number information corresponding to the refueling island.

[0039] In Module Two, the image analysis module, the method for "real-time analysis of decoded video data using a deep learning model to identify cash-receiving behavior" is as follows: The on-site equipment decodes camera video data in real-time using a chip and transmits the data to the model for analysis. A vehicle recognition algorithm monitors whether vehicles are entering the refueling area. When a vehicle is detected entering the refueling area, a vehicle tracking algorithm tracks the vehicle. Once the vehicle stops at the refueling island, cash-receiving behavior monitoring begins. There are two main scenarios for cash receipts: 1. When the pedestrian detection algorithm detects a customer getting out of a vehicle, the pedestrian tracking algorithm begins tracking and identifying the customer, while the ground cash-receiving recognition algorithm identifies whether there is cash exchange on the ground; 2. The in-vehicle cash exchange algorithm monitors whether there is cash exchange inside the vehicle.

[0040] The cash detection algorithm can also be specifically combined with edge detection of the user's hand information to extract images of suspected cash. The edge detection can employ a wavelet edge detection algorithm.

[0041] Once cash-handling behavior is detected, the regional fuel pump number is obtained based on the regional matching results. The fuel pump number information, cash collection time, images, videos, and other information are sent to the matching module. The vehicle tracking algorithm stops monitoring cash collection behavior in that area after the vehicle leaves.

[0042] Specifically, the "obtaining order information corresponding to cash payment behavior" mentioned in Module 3 (Matching Module) can be achieved by: retrieving refueling order information from the order system via the ROCKET MQ message queue, including information such as the fuel nozzle number, nozzle grabbing information, payment method, and payment amount; and then matching the cash payment time and fuel nozzle number information obtained from the image analysis module in Module 2 with the fuel nozzle number and nozzle grabbing time from the order information to obtain the refueling order information corresponding to the cash payment refueling process.

[0043] The "judgment of matched order information" described in Module 4 is implemented as follows: Cash payment behavior is matched with corresponding order information through Module 3. If the payment method of the order information is a non-cash payment method, then it is determined that the refueling process is associated with cash-out behavior.

[0044] Example 2

[0045] As Figure 1As shown, the entire system comprises four modules: a configuration module, an image analysis module, a matching module, and an early warning module. The image analysis module acquires video stream data from the camera. Alternatively, it can obtain relevant parameter information from the configuration module, including the analysis device IP, camera IP, refueling island area information, and the corresponding fuel nozzle number. Then, it analyzes the image data using a deep learning model. If cash-taking behavior is detected, the relevant information is transmitted to the matching module. The matching module retrieves refueling order information from the order system and matches the cash-taking behavior transmitted from the image analysis module with corresponding order information according to matching rules. The early warning module analyzes the matched order information and, upon confirming the cash-taking behavior, issues an alarm.

[0046] Module 1, the configuration module, is implemented as follows:

[0047] Start the front-end streaming media service, configure the IP addresses of all cameras and analysis devices in the gas station's refueling area on the front-end page, and use the streaming media player on the page to capture the live footage, marking the refueling island area and fuel nozzle number information on the footage.

[0048] Module 2, the implementation process of the image analysis module is as follows: Figure 2 As shown:

[0049] S1 tracks vehicles and pedestrians;

[0050] S2 monitors whether vehicles have entered the refueling island;

[0051] If S3 detects a vehicle entering the refueling island, it will activate the cash collection behavior monitoring for that island and simultaneously monitor whether the vehicle leaves the refueling island.

[0052] S4 monitors whether customers and gas station attendants are exchanging items on the ground or next to the vehicle;

[0053] If a customer and a gas station attendant exchange cash on the ground, the S5 system determines whether it is a cash exchange. If so, it sends the information to the matching module and disables the cash collection monitoring for that island.

[0054] If S6 detects a gas station attendant accepting cash through the car window, it will send the information to the matching module and disable the monitoring of cash-receiving behavior on that island.

[0055] If S7 detects a vehicle leaving the refueling island, it will disable the cash collection monitoring for that island.

[0056] The above process involves vehicle detection, pedestrian detection, vehicle tracking, pedestrian tracking, ground cash collection behavior recognition, vehicle-side cash collection recognition, and identification of workers in work clothes.

[0057] Optionally, when a gas station attendant is detected performing refueling, a proximity identifier equipped with the attendant (which can be a communication unit using Wi-Fi and paired with the refueling nozzle) is used. When the attendant holds the nozzle, the communication management unit in the gas station or refueling island is triggered to identify the attendant's identification information and compare it with the user's image detection information. If the identification information and image detection information do not match, the detection level of the video behavior is increased. Specifically, the management platform controls at least two cameras in the refueling area of ​​the gas station to perform near-far separation detection of refueling behavior and analyzes the video information acquired by the cameras. The analysis and processing algorithm can optionally employ the video analysis and convolutional neural network algorithm described in Example 1.

[0058] Optionally, the vehicle and pedestrian detection models can also be combined with target detection algorithms, the tracking model can be based on the deepsort tracking algorithm, the behavior recognition can be based on the pose estimation algorithm, and the identification of tooling personnel can be based on image classification algorithms.

[0059] The implementation steps for the matching module in Module 3 are as follows:

[0060] 1. Match cash collection orders. Based on the cash collection time, match the order pickup / hang-up time period, and match all orders that meet the conditions: a. If the cash collection time is within the pickup / hang-up time period, match directly; b. If the cash collection time is not within any order pickup / hang-up time period, check if the cash collection time is within 15 seconds before or after the pickup / hang-up time of a certain order; 2. Filter all orders not from this refueling island; 3. Fill in the orders. If multiple orders are matched, check if the order numbers are consecutive. If not, fill in all the orders in between.

[0061] The implementation method for Module 4, the early warning module, is as follows:

[0062] The cash payment behavior is matched with the corresponding order information through Module 3. If the order information shows that the payment method is a non-cash payment method, it is determined that the refueling process has resulted in cash-out behavior. The cash payment image, video, gun-collecting person's ID information, and order information are then transmitted to the management platform.

[0063] Based on the examples described above, the features involving method steps in one embodiment can be implemented by a computer device / system provided by the present invention, the computer device / system including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the methods described in the above embodiments.

[0064] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium. For example, in the embodiments of the present invention, the program can be stored in the storage medium of a computer system and executed by at least one processor in the computer system to implement the processes including the embodiments of the video playback methods described above. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0065] Accordingly, a storage medium is also provided that stores a computer program thereon, wherein the program, when executed by a processor, implements any of the method steps involved in the above embodiments.

[0066] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0067] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A multi-linkage gas station cash-out behavior identification system, characterized in that: The system comprises four modules: a configuration module, an image analysis module, a matching module, and an alarm module. The configuration module acquires images from the gas station's on-site cameras and marks the refueling island area, the corresponding fuel nozzle number, and the ID information of the person picking up the nozzle in the image. There are at least two cameras. The image analysis module uses a deep learning model to analyze the decoded video data in real time and identify cash-receiving behavior. The matching module obtains order information from the gas station's order system and information confirming that a cash-collection behavior has occurred from the image analysis module. It then matches the fuel nozzle number and cash-collection time uploaded by the image analysis module with the fuel nozzle number and nozzle lifting / hanging time in the order information to obtain the order information corresponding to the cash-collection behavior. The alarm module judges the matched order information, and when it detects cash-out behavior, it transmits the alarm information to the gas station's management platform. The alarm information includes at least order information, the ID of the person who picked up the gun, and video image information of the person who picked up the gun; The configuration module acquires the on-site camera footage of the gas station at the front end and marks the refueling island area, the corresponding fuel nozzle number information, and the ID information of the person picking up the nozzle in the footage. This includes: starting the front-end streaming media service, configuring the camera IP address, acquiring the on-site footage using the streaming media RTSP protocol, marking the refueling island area based on the canvas API, and marking the fuel nozzle number information corresponding to the refueling island. The image analysis module describes the use of a deep learning model to analyze and decode video data in real time to identify cash collection behavior. This includes: the on-site equipment decodes camera video data in real time through a chip, transmits the data to a neural network analysis model for analysis, tracks the vehicle after it is detected that it has entered the refueling area, and starts monitoring cash collection behavior after it is detected that the vehicle has entered the refueling island and stopped. The monitoring of cash collection behavior includes: after a customer gets off the vehicle, the system begins to track and identify the customer to determine whether there is any cash exchange on the ground or inside the vehicle; specifically, the identification of cash exchange on the ground is performed using a neural network analysis model to detect the hand movements of the person carrying the cash and the customer, obtain an image of the hand quadrilateral, and compare it with a preset cash feature image.

2. The gas station cash-out behavior identification system as described in claim 1, characterized in that: The matching module's acquisition of order information corresponding to cash-receiving behavior includes: acquiring refueling order information from the order system via the ROCKET MQ message queue, including the fuel nozzle number, nozzle grabbing information, payment method, and payment amount; and matching the cash-receiving time and fuel nozzle number information acquired through the image analysis module with the fuel nozzle number and nozzle grabbing time from the order information to obtain the refueling order information corresponding to the cash-receiving refueling process.

3. The multi-linkage gas station cash-out behavior identification system as described in claim 2, characterized in that: A neural network analysis model is used to analyze and detect the hand movements of the person carrying the gun and the customer, and to obtain the area image of the quadrilateral of the hand. The acquisition of the quadrilateral image information of the hand is based on the transformation of the logo image in the gas station and then compared with the cash feature image to determine whether it is a cash image.

4. The multi-linkage gas station cash-out behavior identification system as described in claim 3, characterized in that: Based on the location information of the signage image set at the gas station and the angle information of the camera, a deformation matrix is ​​preset. When extracting the quadrilateral information image of the user's hand, if the preset deformation matrix exists, the deformation transformation is directly performed and then compared with the standard image; if the preset deformation matrix does not exist, the deformation matrix is ​​calculated based on the signage image and the signage image obtained from the camera.

5. The system for identifying cash-out behavior at gas stations with multiple linkages as described in claim 4, characterized in that: The neural network analysis module includes at least one convolutional layer, at least one fully connected layer, and a softmax layer; the feature map of the last convolutional layer is input into the at least one fully connected layer to obtain a two-dimensional vector output by the last fully connected layer in the at least one fully connected layer.

Images