Grey-box-model-driven convenience store assistant service specification identification system and method
By applying a gray box model-driven identification system in the gas station convenience store, combining a lightweight target detection model and an interpretable physical information excitation algorithm, the problem of identifying irregular service behaviors of clerks in the existing technology is solved, and video inspection with high accuracy and low cost is achieved, replacing manpower and reducing management costs.
Patent Information
- Application Number
- CN202411646862.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-18
- Publication Date
- 2025-03-04
AI Technical Summary
The prior art is difficult to quickly and accurately identify irregular service behaviors of clerks in complex gas station convenience stores, resulting in high inspection costs and low efficiency.
The convenience store clerk service specification recognition system driven by a gray box model is used, combined with the lightweight object detection model Yol OV5 and the human action habit algorithm based on interpretable physical information, and the identification of irregular service behaviors is achieved through video inspection strategies, human body element detection and physical prior knowledge.
It has achieved standardized identification of high accuracy and low economic costs in the video surveillance service industry employees under poor network environment, poor hardware facilities, poor monitoring environment, and complex scenarios, and replaced human resources to complete standardized video inspection of convenience store clerks, reducing enterprise management costs.
Smart Images
Figure CN119559698A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of store clerk service specification recognition, and in particular to a gray box model-driven convenience store clerk service specification recognition system and method. Background Art
[0002] The development of artificial intelligence technology has promoted the intelligent transformation and upgrading of various industries. With the widespread application of deep learning algorithms in the field of video intelligent analysis, the manual convenience store service standard video inspection method has gradually shifted to a machine intelligent method. Taking the service action standard recognition scenario of gas station convenience store employees as an example, the inspection personnel generally use video playback to check whether the store staff has irregular behaviors such as customers entering the store without waving, handing over items with both hands, customers leaving the store without waving, and store staff sleeping on the job. This method increases the cost of manual inspections on the one hand, and also increases the incompleteness of inspections caused by manual fatigue. At present, with the help of computer vision technologies such as deep learning, the service standards of convenience store employees are analyzed in real time and automatically warned, and the irregular service behaviors of the store staff are recorded and the current video clips are saved for manual re-inspection.
[0003] Due to the complex on-site environment of gas station convenience stores, different camera installation locations and lighting conditions, and other practical problems, in addition, the clerk's service actions generally require comprehensive judgment of abnormal behavior through video analysis. This poses severe challenges to the AI inspection speed, the recognition accuracy and recall rate of irregular service behaviors. Therefore, we proposed a gray-box model-driven convenience store clerk service standard recognition system and method. Summary of the invention
[0004] The purpose of the present invention is to provide a gray box model driven convenience store clerk service specification identification system and method to solve the problems raised in the above background technology.
[0005] To achieve the above object, the present invention provides the following technical solutions:
[0006] A gray-box model-driven convenience store clerk service standard recognition system includes irregular service actions, wherein the output end of the irregular service actions is electrically connected to a video inspection strategy, the output end of the video inspection strategy is electrically connected to a human element detection, the output end of the human element detection is electrically connected to a human action habit algorithm, and the input end of the human action habit algorithm is electrically connected to physical prior knowledge.
[0007] A gray box model driven convenience store clerk service standard identification method, the non-standard service action comprises the following steps:
[0008] S1. Abstract the service standard requirements of gas station convenience stores into an achievable AI problem. Non-standard service actions include: employees not waving when customers enter the store, employees not waving when customers leave the store, employees not handing over items with both hands, and employees sleeping on duty.
[0009] As a preferred solution of the present invention, the video inspection strategy includes the following steps:
[0010] S2. Connect the cameras to the visual networking platform, analyze the inspection tasks that are suitable for each camera, and map the camera and irregular service action task association mechanism to the parameter configuration management module of this platform through software development to facilitate later maintenance;
[0011] S3. According to the inspection time requirements, the camera scheduled inspection task mechanism is mapped to the parameter configuration management module of this platform through software development. Relevant personnel can set the scheduled task of automatic inspection for each irregular service action through the parameter management module of this platform;
[0012] S4. Realize message connection between the backend and AI model through asynchronous mechanism.
[0013] As a preferred solution of the present invention, the human element detection comprises the following steps:
[0014] S5. adopts the lightweight target detection model yo l ov5 and innovatively designs the detection categories, and combines explainable physical information of human actions as prior knowledge.
[0015] As a preferred solution of the present invention, the physical prior knowledge includes the following steps:
[0016] S6. The prior knowledge that exists in the service process is: there is and must be a condition where employees and customers are in the convenience store at the same time, that is, the target categories "blue work clothes" and "non-employees" exist at the same time;
[0017] S7. The prior knowledge in the process of entering the store is that the customer is close to the cashier counter and the clerk is in the store, that is, the target categories "blue work clothes" and "non-employee" exist at the same time, and the coordinate position of "non-employee" in the continuous multi-frame image is constantly approaching the cashier counter;
[0018] S8. The prior knowledge in the process of leaving the store is that the customer is away from the cashier counter and the clerk is in the store, that is, the target categories "blue work clothes" and "non-employee" exist at the same time, and the coordinate position of "non-employee" in the continuous multi-frame image is constantly away from the cashier counter;
[0019] S9. The prior knowledge in the object handing process is: two people are close and the customer has both hands, that is, the target categories "blue work clothes" and "non-employee" exist at the same time, and the position coordinates of both are lower than the specified threshold and there is one of the "double empty hands" or "double hands holding things" targets;
[0020] S10. The prior knowledge that exists during the sleeping process is that the clerk is stationary and not standing for a long time, that is, the target category "blue work clothes" exists and the position fluctuation of the target in multiple consecutive frames of images is lower than the specified threshold and the target height does not exceed the specified width threshold.
[0021] As a preferred solution of the present invention, the human action habit algorithm comprises the following steps:
[0022] S11. The pseudo algorithm for identifying the action of entering a store without waving is designed as follows:
[0023]
[0024]
[0025] S12. The pseudo algorithm for identifying the action of leaving a store without waving is as follows:
[0026]
[0027] S13. The pseudo algorithm for identifying the hand-over action is designed as follows:
[0028]
[0029]
[0030] S14. The pseudo algorithm for identifying the action of sleeping on duty is designed as follows:
[0031]
[0032] Compared with the prior art, the present invention has the following beneficial effects:
[0033] 1. In the present invention, the technical problems caused by high cloud network costs and complex scenarios in the visual network scenario to the realization of artificial intelligence model technology are solved through this technical solution, which aims to intelligently identify the service standardization of employees in the video surveillance service industry under conditions of poor network environment, poor hardware facilities, poor monitoring environment and complex scenarios. On the one hand, it helps to replicate and promote this method to video surveillance industries such as supermarkets and retail industries. At the same time, the low economic cost and high accuracy solution proposed in this solution can replace manpower to a certain extent to complete the video inspection of convenience store clerk service standards, assist in the supervision of gas station convenience store services and reduce the management costs of enterprises.
[0034] 2. In the present invention, irregular service behaviors such as customers entering the store without waving, handing over items without both hands, customers leaving the store without waving, and store clerks sleeping on the job can be solved. This method combines the lightweight target detection algorithm yo l ov5 and the human action habit algorithm based on interpretable physical information. On the one hand, this interpretable method has a high accuracy rate, and on the other hand, it does not require massive data to learn model parameters, so it has better robustness. At the same time, it does not require GPU acceleration and can quickly make prediction results. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 This is a framework diagram of the identification system of the present invention;
[0036] Figure 2 This is an example diagram of the human body element detection category annotation of the present invention;
[0037] Figure 3 It is a physical prior knowledge representation diagram for judging the behavior of the clerk not waving in the present invention;
[0038] Figure 4 This is a physical prior knowledge representation diagram for judging the behavior of store clerks sleeping on the job in the present invention;
[0039] Figure 5 This is a warning image of a customer entering a convenience store without waving;
[0040] Figure 6 This is the convenience store sleeping alarm diagram of the present invention.
[0041] In the figure: 1. Irregular service actions; 2. Video inspection strategy; 3. Human element detection; 4. Physical prior knowledge; 5. Human action habit algorithm. DETAILED DESCRIPTION
[0042] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0043] For examples, see Figure 1-Figure 6 , the present invention provides a technical solution:
[0044] A gray-box model-driven convenience store clerk service standard recognition system includes an irregular service action 1, the output end of the irregular service action 1 is electrically connected to a video inspection strategy 2, the output end of the video inspection strategy 2 is electrically connected to a human element detection 3, the output end of the human element detection 3 is electrically connected to a human action habit algorithm 5, and the input end of the human action habit algorithm 5 is electrically connected to physical prior knowledge 4.
[0045] A gray box model driven convenience store clerk service standard identification method, non-standard service action 1 includes the following steps:
[0046] S1. Abstract the service standard requirements of gas station convenience stores into an achievable AI problem. Non-standard service actions include: employees not waving when customers enter the store, employees not waving when customers leave the store, employees not handing over items with both hands, and employees sleeping on duty.
[0047] Video inspection strategy 2 includes the following steps:
[0048] S2. Connect the cameras to the visual networking platform, analyze the inspection tasks that are suitable for each camera, and map the camera and irregular service action task association mechanism to the parameter configuration management module of this platform through software development to facilitate later maintenance;
[0049] S3. According to the inspection time requirements, the camera scheduled inspection task mechanism is mapped to the parameter configuration management module of this platform through software development. Relevant personnel can set the scheduled task of automatic inspection for each irregular service action through the parameter management module of this platform;
[0050] S4. Realize message connection between the backend and AI model through asynchronous mechanism.
[0051] Human element detection 3 includes the following steps:
[0052] S5. adopts the lightweight target detection model yo l ov5 and innovatively designs the detection categories, and combines explainable physical information of human actions as prior knowledge.
[0053] Specifically, the English and Chinese labels of the human element detection category are as follows:
[0054] Chinese label English label Blue work clothes blue_cloth Non-employees person Single empty hand one_empty_hand Holding things with one hand one_hold_hand Double empty hands two_empty_hand Holding things with both hands two_hold_hand Head head
[0055] Among them, the target detection model is responsible for detecting all human target objects that represent irregular service behaviors. This method only uses a lightweight multi-scale target detection model to realize all the required human target objects that can be used to realize irregular service behaviors, which greatly reduces the computing power cost required for the model scale and improves the effectiveness of inspections. (2): Human action habit algorithm 5 integrates physical prior knowledge into human target objects such as employee bodies, non-employee bodies, single and double hands, and video sequence frame image analysis, and realizes the AI function of irregular service recognition for store clerks, which is extremely challenging for visual model technology, with an interpretable and verifiable algorithm design. This interpretable method has a high accuracy rate on the one hand, and does not require massive data to learn model parameters on the other hand, so it has better robustness. At the same time, it does not require GPU acceleration and can quickly make prediction results. Obviously, it has strong technical replicability and promotion for video behavior analysis under conditions of poor network environment, poor hardware facilities, poor monitoring environment, and complex scenes.
[0056] In this embodiment, Figure 4 , Figure 5 and Figure 6 As shown, the physical prior knowledge 4 includes the following steps:
[0057] S6. The prior knowledge that exists in the service process is: there is and must be a condition where employees and customers are in the convenience store at the same time, that is, the target categories "blue work clothes" and "non-employees" exist at the same time;
[0058] S7. The prior knowledge in the process of entering the store is that the customer is close to the cashier counter and the clerk is in the store, that is, the target categories "blue work clothes" and "non-employee" exist at the same time, and the coordinate position of "non-employee" in the continuous multi-frame image is constantly approaching the cashier counter;
[0059] S8. The prior knowledge in the process of leaving the store is that the customer is away from the cashier counter and the clerk is in the store, that is, the target categories "blue work clothes" and "non-employee" exist at the same time, and the coordinate position of "non-employee" in the continuous multi-frame image is constantly away from the cashier counter;
[0060] S9. The prior knowledge in the object handing process is: two people are close and the customer has both hands, that is, the target categories "blue work clothes" and "non-employee" exist at the same time, and the position coordinates of both are lower than the specified threshold and there is one of the "double empty hands" or "double hands holding things" targets;
[0061] S10. The prior knowledge that exists during the sleeping process is that the clerk is stationary and not standing for a long time, that is, the target category "blue work clothes" exists and the position fluctuation of the target in multiple consecutive frames of images is lower than the specified threshold and the target height does not exceed the specified width threshold.
[0062] The human action habit algorithm 5 includes the following steps:
[0063] S11. The pseudo algorithm for identifying the action of entering a store without waving is designed as follows:
[0064]
[0065]
[0066] S12. The pseudo algorithm for identifying the action of leaving a store without waving is as follows:
[0067]
[0068] S13. The pseudo algorithm for identifying the hand-over action is designed as follows:
[0069]
[0070]
[0071] S14. The pseudo algorithm for identifying the action of sleeping on duty is designed as follows:
[0072]
[0073] The workflow of the present invention: When the gray box model designed in this scheme is used to drive the convenience store clerk service standard recognition system and method, the irregular service actions 1 include customers entering the store without waving, handing over items without both hands, customers leaving the store without waving, and clerks sleeping on duty. For such inspection tasks, the video inspection strategy 2 module is responsible for the access of multiple cameras on site, the timing inspection task mechanism, the camera and task association and other functions. Then, after the inspection strategy is issued, the human element detection 3 model is requested to obtain the target category and location information, where the detection capability is trained by the detection category data designed by the scheme. Then, the human action habit algorithm 5 is implemented based on the physical information knowledge module, and after analysis, it is asynchronously returned to the requester and an early warning is issued through a pop-up window on the monitoring screen.
[0074] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A gray box model driven convenience store clerk service standard recognition system, including irregular service actions (1), characterized in that: The output end of the irregular service action (1) is electrically connected to a video inspection strategy (2), the output end of the video inspection strategy (2) is electrically connected to a human element detection (3), the output end of the human element detection (3) is electrically connected to a human action habit algorithm (5), and the input end of the human action habit algorithm (5) is electrically connected to physical prior knowledge (4).
2. A gray-box model driven convenience store clerk service standard recognition method, applicable to the gray-box model driven convenience store clerk service standard recognition system described in claim 1, comprising irregular service actions (1), video inspection strategy (2), human element detection (3), physical prior knowledge (4) and human action habit algorithm (5): characterized in that: The irregular service action (1) comprises the following steps: S1. Abstract the service standard requirements of gas station convenience stores into an achievable AI problem. Non-standard service actions include: employees not waving when customers enter the store, employees not waving when customers leave the store, employees not handing over items with both hands, and employees sleeping on duty.
3. The method for identifying service standards of convenience store employees driven by a gray box model according to claim 2 is characterized by: The video inspection strategy (2) comprises the following steps: S2. Connect the cameras to the visual networking platform, analyze the inspection tasks that are suitable for each camera, and map the camera and irregular service action task association mechanism to the parameter configuration management module of this platform through software development to facilitate later maintenance; S3. According to the inspection time requirements, the camera scheduled inspection task mechanism is mapped to the parameter configuration management module of this platform through software development. Relevant personnel can set the scheduled task of automatic inspection for each irregular service action through the parameter management module of this platform; S4. Realize message connection between the backend and AI model through asynchronous mechanism.
4. The method for identifying service standards of convenience store employees driven by a gray box model according to claim 2 is characterized by: The human element detection (3) comprises the following steps: S5. It adopts the lightweight target detection model yolov5 and innovatively designs the detection categories, and combines interpretable physical information of human actions as prior knowledge.
5. The method for identifying convenience store clerk service standards driven by a gray box model according to claim 2 is characterized by: The physical prior knowledge (4) comprises the following steps: S6. The prior knowledge that exists in the service process is: there is and must be a condition where employees and customers are in the convenience store at the same time, that is, the target categories "blue work clothes" and "non-employees" exist at the same time; S7. The prior knowledge in the process of entering the store is that the customer is close to the cashier counter and the clerk is in the store, that is, the target categories "blue work clothes" and "non-employee" exist at the same time, and the coordinate position of "non-employee" in the continuous multi-frame image is constantly approaching the cashier counter; S8. The prior knowledge in the process of leaving the store is that the customer is away from the cashier counter and the clerk is in the store, that is, the target categories "blue work clothes" and "non-employee" exist at the same time, and the coordinate position of "non-employee" in the continuous multi-frame image is constantly away from the cashier counter; S9. The prior knowledge in the object delivery process is: two people are close to each other and the customer has both hands, that is, the target categories "blue work clothes" and "non-employee" exist at the same time, and the position coordinates of both are lower than the specified threshold and there is one of the "empty hands" or "two hands holding things" targets; S10. The prior knowledge that exists during the sleeping process is that the clerk is stationary and not standing for a long time, that is, the target category "blue work clothes" exists and the position fluctuation of the target in multiple consecutive frames of images is lower than the specified threshold and the target height does not exceed the specified width threshold.
6. The method for identifying service standards of convenience store employees driven by a gray box model according to claim 2, characterized in that: The human action habit algorithm (5) comprises the following steps: S11. The pseudo algorithm for identifying the action of entering a store without waving is designed as follows: S12. The pseudo algorithm for identifying the action of leaving a store without waving is as follows: S13. The pseudo algorithm for identifying the hand-over action is designed as follows: S14. The pseudo algorithm for identifying the action of sleeping on duty is designed as follows: