Cashier abnormal behavior detection and association method and system, medium and program product

Abstract

The invention discloses a cashier abnormal behavior detection and association method and system, a medium and a program product, and the method comprises the steps: receiving a real-time monitoring video and real-time order information; determining a sold scanned commodity and a sold weighed commodity, and determining a scanning state and a weighing state; determining a current settlement commodity according to the real-time order information; determining a currently processed commodity corresponding to the current time point; determining a wrong scanning state; judging whether an abnormal behavior occurs or not according to the scanning state, the weighing state and the wrong scanning state; if the abnormal behavior occurs, determining an abnormal sold article and an abnormal monitoring fragment corresponding to the abnormal behavior; and associating the real-time order information of the abnormally sold article with the abnormal monitoring fragment to obtain cashier abnormal information, and sending the cashier abnormal information to the supervision terminal and the cashier device. The method and the device are used for improving the efficiency and the accuracy of detecting abnormal cashier behaviors and further improving the work efficiency of cashier loss prevention.

Description

Technical Field

[0001] The present application belongs to the field of cashier loss prevention, and in particular, relates to a cashier abnormal behavior detection and association method, system, medium and program product. Background Art

[0002] The retail industry has long faced a serious loss problem. These losses come from a variety of sources, including but not limited to employee fraud, customer theft, supply chain vulnerabilities, etc. According to statistics, these losses cause tens of billions of dollars in economic losses to the global retail industry each year. This persistent problem has prompted retailers to seek more effective solutions.

[0003] With the rapid development of information technology, especially breakthroughs in computer vision, big data analysis and artificial intelligence, new technical means have been provided for abnormal behavior detection. These technological advances have made more complex and accurate detection methods possible.

[0004] In the related technologies, the mainstream method in the field of anomaly detection is based on big data analysis. Retailers analyze massive amounts of POS transaction data, build normal behavior models, and then use these models to identify abnormal transactions that deviate from normal patterns. This method can process large-scale data and discover subtle anomalies that are difficult to detect with traditional methods.

[0005] However, the above technical means are based on big data analysis based on surveillance videos. Due to the large flow of people in supermarkets and the huge amount of surveillance video data, the detection method of big data analysis may have the problem of reduced accuracy and its detection efficiency. In addition, it may be difficult to determine the product information involved in abnormal cashier behavior from the surveillance video, and the staff may need to re-inventory the products, which reduces the efficiency of cashier loss prevention. Summary of the invention

[0006] The present application provides a cashier abnormal behavior detection and association method, system, medium and program product for improving the efficiency and accuracy of cashier abnormal behavior detection, thereby improving the work efficiency of cashier loss prevention.

[0007] In a first aspect, the present application provides a cashier abnormal behavior detection and association method, which receives real-time monitoring video sent by a monitoring device and real-time order information sent by a cashier device; Determine the sold scanned goods and the sold weighed goods according to the real-time monitoring video, and determine the scanning status of the sold scanned goods and the weighing status of the sold weighed goods, where the scanning status includes scanned and unscanned, and the weighing status includes weighed and unweighed; Determine the current settlement product according to the real-time order information, where the current settlement product includes the current settlement product information and the current time point; Determine the currently processed commodity corresponding to the current time point in the real-time monitoring video, where the currently processed commodity includes currently processed commodity information; Determine the wrong scanning status according to the current settlement product information and the pre-processed product information. The wrong scanning status includes confirmed wrong scanning and not wrong scanning. When the current settlement product information and the pre-processed product information are the same, the wrong scanning status is not wrong scanning. When the current settlement product information and the pre-processed product information are different, the wrong scanning status is confirmed wrong scanning. Determine whether abnormal behavior occurs according to the scanning status, weighing status and wrong scanning status, and confirm that abnormal behavior occurs when one or more of the scanning status is not scanned, the weighing status is not weighed, and the wrong scanning status is confirmed wrong scanning is met; If abnormal behavior occurs, the abnormal sold items and abnormal monitoring clips corresponding to the abnormal behavior are determined; The real-time order information of abnormally sold items is associated with the abnormal monitoring fragment to obtain the cashier abnormality information, and the cashier abnormality information is sent to the supervision terminal and the cashier device.

[0008] By adopting the above technical solution, it is possible to receive the monitoring video sent by the monitoring device and the order information sent by the cash register device in real time, providing a sufficient data basis for subsequent abnormal behavior detection. Secondly, by analyzing the real-time monitoring video, it is possible to accurately identify the scanned goods sold and the weighed goods sold, and judge their scanning status and weighing status, providing key information for detecting abnormal behavior. According to the real-time order information, the current settlement goods are determined, and compared with the current processed goods in the real-time video, it is possible to effectively judge whether there is a wrong scanning behavior. Finally, by comprehensively considering the scanning status, weighing status and wrong scanning status, it is possible to accurately judge whether there is abnormal cash register behavior. Once an abnormality is found, the system will automatically extract the order information of the abnormal sold items and the related abnormal monitoring clips, and associate the two to generate cash register abnormal information, and send it to the supervision terminal and cash register device at the same time, realizing real-time early warning and prevention of abnormal behavior. This method greatly improves the detection rate and processing efficiency of cash register abnormal behavior through multi-dimensional information fusion and intelligent analysis, effectively prevents losses in the cash register link, improves the efficiency and accuracy of cash register abnormal behavior detection, and thus improves the efficiency of cash register loss prevention.

[0009] In conjunction with some embodiments of the first aspect, in some embodiments, determining the sold scanned goods and the sold weighed goods according to the real-time monitoring video specifically includes: Perform denoising and contrast adjustment on the real-time monitoring video to obtain a pre-processed monitoring video; Use object detection algorithms to identify items for sale in preprocessed surveillance videos; Determine the sold goods containing barcodes as sold scanned goods; The sold goods without a barcode are determined as sold weighed goods.

[0010] By adopting the above technical solutions, the real-time monitoring video is first pre-processed by denoising and contrast adjustment to improve the video quality, laying the foundation for subsequent commodity identification. Secondly, the method uses advanced target detection algorithms to quickly and accurately identify all sold commodities from the video, overcoming the limitations of human eye observation. Thirdly, the method intelligently determines whether the commodity contains a barcode and whether it is a sold scanned commodity or a sold weighed commodity, thereby realizing the automatic classification of commodity categories. This method makes full use of computer vision and artificial intelligence technology to realize the automatic identification and classification of sold commodities, which not only greatly reduces labor costs and improves inspection efficiency, but also ensures the accuracy and fairness of the identification results through objective and consistent algorithm rules, providing a reliable data reference for subsequent abnormal behavior analysis.

[0011] In conjunction with some embodiments of the first aspect, in some embodiments, determining the scanning status of the scanned goods sold and the weighing status of the weighed goods sold specifically includes: Determine the sold scanning order information and sold weighing order information corresponding to the sold scanning goods and the sold weighing goods according to the real-time monitoring video; Obtain the processed order information entered by the cash register device; When it is determined that the processed order information includes the sold scan order information, the scanning status of the sold scan commodity corresponding to the sold scan order information is determined as scanned; When it is determined that the processed order information does not include the sold scanned order information, the scanning status of the sold scanned commodity corresponding to the sold scanned order information is determined as unscanned; When it is determined that the processed order information includes the sold weighed order information, the weighing status of the sold weighed commodity corresponding to the sold weighed order information is determined as weighed; When it is determined that the processed order information does not include the sold weighed order information, the weighing status of the sold weighed commodity corresponding to the sold weighed order information is determined as unweighed.

[0012] By adopting the above technical solution, the real-time monitoring video and the order information of the cash register equipment are utilized. Through the intelligent matching and comparison of the two, the real processing status of each sold commodity can be determined efficiently and accurately. First, the order information corresponding to the sold scanned goods and weighed goods is extracted from the real-time video, and then the processed orders entered by the cash register equipment are obtained. By comparing the two types of order information, it can be determined whether the sold goods identified in the video have been scanned or weighed in the cash register system. This audit method combining video and order, on the one hand, makes up for the shortcomings of a single data dimension and realizes multi-angle and all-round status determination; on the other hand, it avoids the subjectivity and differences of manual verification and ensures the consistency and authority of the determination results. Through automated and precise status determination, this method can detect omissions or erroneous behaviors in the cash register process in the first time, providing a basis for abnormal warning and problem rectification, which is conducive to improving cash register efficiency, standardizing cash register processes, and reducing business risks. It has a positive effect on maintaining supermarket interests and improving customer satisfaction.

[0013] In combination with some embodiments of the first aspect, in some embodiments, real-time order information of abnormally sold items is associated with abnormal monitoring segments to obtain cashier abnormality information, which specifically includes: The real-time order information is superimposed on the corresponding abnormal monitoring segment association through the OSD character superposition method to obtain the cashier abnormal information; Generate a viewing control and set the viewing control at a preset position of cashier exception information; In response to the user clicking the view control, the abnormal monitoring clip is played.

[0014] By adopting the above technical solution, OSD character overlay technology is used to overlay the order information of abnormally sold items into the corresponding abnormal monitoring video clips in real time, so that the abnormal information is perfectly associated with the abnormal process. When the manager views the cashier abnormal information, he can clearly see the sales details of each abnormal item. At the same time, the method also provides a highly humanized interactive function. By setting a viewing control on the cashier abnormal information interface, when the user clicks the control, the system will automatically retrieve and play the corresponding abnormal video clip, so that the user can go directly to the abnormal scene with one click and quickly and intuitively review the full picture of the abnormal behavior. This one-click access to abnormal information presentation method greatly improves the efficiency and experience of cashier abnormal management. Managers no longer need to manually search or match abnormal information and videos, nor do they need to switch between multiple systems, thus saving a lot of time and energy. At the same time, through the video playback function, managers can repeatedly observe and carefully analyze each abnormal detail, which helps to comprehensively find the root cause of the problem and formulate targeted improvement measures.

[0015] In combination with some embodiments of the first aspect, in some embodiments, after generating a viewing control and setting the viewing control at a preset position of the cashier exception information, the method further includes: Determine the location information of the abnormally sold items based on the abnormal monitoring clips; An abnormal object marking frame is generated according to the position information, and the center point of the abnormal object marking frame is the same as the center point of the position information.

[0016] By adopting the above technical solution, firstly, by analyzing the abnormal monitoring video clips, the position coordinate information of the abnormal sold items in the picture is intelligently extracted. Based on this, the system automatically generates an abnormal item marking frame and accurately draws it at the actual position of the abnormal item. It is worth mentioning that the center point of the marking frame completely coincides with the center point of the abnormal item, ensuring the accuracy of the identification. Through this visual marking method, the abnormal items are clearly highlighted in the video screen, and the manager can quickly focus on the key items where the abnormal behavior occurs without viewing them frame by frame or manually identifying them. This not only greatly saves the time and energy of abnormal audits, but also reduces the difficulty of identifying non-obvious abnormal behaviors. At the same time, the accurate positioning of abnormal items also provides a direct basis for subsequent responsibility identification and problem tracing, which helps managers accurately analyze the causes of abnormalities and formulate corrective measures in a timely manner. Therefore, the introduction of the abnormal item marking frame greatly improves the effect and efficiency of abnormal information presentation, makes abnormal management more accurate and efficient, and provides strong support for optimizing the cashier process and standardizing cashier banking.

[0017] In conjunction with some embodiments of the first aspect, in some embodiments, after associating the real-time order information of the abnormally sold items with the abnormal monitoring clips to obtain the abnormal cashier information, and sending the abnormal cashier information to the supervision terminal and the cashier device, the method further includes: Identify the abnormal staff members who handled the abnormal sold items in the abnormal monitoring footage; Determine whether to send a warning message to the cashier terminal corresponding to the abnormal staff member according to the preset setting, and the warning message includes the real-time order information of the abnormally sold items; Obtain relevant information of abnormal staff and send it to the supervision terminal.

[0018] By adopting the above technical solution, the abnormal monitoring fragments are intelligently analyzed to accurately identify the direct person responsible for each abnormal behavior, that is, the abnormal staff. The abnormal information is accurately associated with the specific staff, making the abnormal traceability more specific and in place. On this basis, the method automatically determines whether it is necessary to send a warning reminder to the staff involved according to the preset rules. Once the warning condition is triggered, the system will push the warning information containing the details of the abnormal sold items to the cash register terminal of the corresponding staff in real time. This point-to-point abnormal reminder method ensures the accurate delivery of problem information, allowing front-line staff to realize their mistakes at the first time, so as to quickly take remedial measures to minimize the impact of the abnormality. At the same time, the method will also report the relevant information of the abnormal staff to the supervision terminal, providing data support for managers to fully control the abnormal situation and carry out subsequent assessment and accountability. By accurately locking the circulation scope of abnormal information within the scope of relevant responsible persons, it not only improves the timeliness of abnormal handling, but also avoids the abuse and interference of information release.

[0019] In conjunction with some embodiments of the first aspect, in some embodiments, after associating the real-time order information of the abnormally sold items with the abnormal monitoring clips to obtain the abnormal cashier information, and sending the abnormal cashier information to the supervision terminal and the cashier device, the method further includes: If no abnormal behavior occurs, the real-time monitoring video and real-time order information are marked as qualified operations; Upload real-time surveillance video and real-time order information to the preset database.

[0020] By adopting the above technical solution, the system can automatically determine whether each cashier operation is standardized and qualified through intelligent analysis of real-time video and order information. For operations without abnormal behavior, the method will mark them as qualified and clearly distinguish between normal cashier behavior and abnormal cashier behavior. This clear behavioral distinction helps managers accurately evaluate the performance of cashiers and objectively understand and judge the overall compliance of the cashier process. At the same time, the method will also archive all real-time monitoring videos and order information and upload them to the designated database. Through standardized and systematic data storage, it is convenient for subsequent query and traceability, and also lays a data foundation for in-depth applications such as big data analysis and behavior modeling. Based on complete and real historical data, managers can summarize common problems in cashier work, characterize benchmark behavior characteristics, and provide decision-making basis for optimizing training programs and improving management measures. From a higher level, by continuously accumulating and analyzing massive cashier data, it helps companies gain insight into sales rules, optimize marketing strategies, and improve customer experience.

[0021] In the second aspect, an embodiment of the present application provides a cashier abnormal behavior detection and association system, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and one or more processors call the computer instructions to enable the system to execute the method described in the first aspect and any possible implementation method of the first aspect.

[0022] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions, which, when executed on a system, causes the system to execute the method described in the first aspect and any possible implementation of the first aspect.

[0023] In a fourth aspect, an embodiment of the present application provides a computer program product, characterized in that when the computer program product runs on a system, the system executes the method described in any possible implementation manner in the first aspect.

[0024] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. The present application provides a cashier abnormal behavior detection and association method, which can receive the monitoring video sent by the monitoring device and the order information sent by the cashier device in real time, and provide a sufficient data basis for subsequent abnormal behavior detection. Secondly, by analyzing the real-time monitoring video, it is possible to accurately identify the sold scanned goods and the sold weighed goods, and judge their scanning status and weighing status, which provides key information for detecting abnormal behavior. According to the real-time order information, the current settlement goods are determined, and compared with the current processed goods in the real-time video, it can be effectively judged whether there is a wrong scanning behavior. Finally, considering the scanning status, weighing status and wrong scanning status, it is possible to accurately judge whether there is cashier abnormal behavior. Once an abnormality is found, the system will automatically extract the order information of the abnormal sold items and the related abnormal monitoring clips, and associate the two to generate cashier abnormal information, and send it to the supervision terminal and cashier device at the same time, realizing real-time early warning and prevention of abnormal behavior. This method greatly improves the detection rate and processing efficiency of cashier abnormal behavior through multi-dimensional information fusion and intelligent analysis, effectively prevents losses in the cashier link, improves the efficiency and accuracy of cashier abnormal behavior detection, and thus improves the efficiency of cashier loss prevention.

[0025] 2. The present application provides a cash register abnormal behavior detection and association method, which utilizes the real-time monitoring video and the order information of the cash register device. Through the intelligent matching and comparison of the two, the real processing status of each sold commodity can be determined efficiently and accurately. First, the order information corresponding to the sold scanned goods and weighed goods is extracted from the real-time video, and then the processed order entered by the cash register device is obtained. By comparing the two types of order information, it can be determined whether the sold goods identified in the video have been scanned or weighed in the cash register system. This audit method combining video and order, on the one hand, makes up for the shortcomings of a single data dimension and realizes multi-angle and all-round status determination; on the other hand, it avoids the subjectivity and differences of manual verification and ensures the consistency and authority of the determination results. Through automated and precise status determination, this method can detect omissions or erroneous behaviors in the cash register process in the first time, providing a basis for abnormal warning and problem rectification, which is conducive to improving cash register efficiency, standardizing cash register processes, and reducing business risks. It has a positive effect on maintaining supermarket interests and improving customer satisfaction.

[0026] 3. The present application provides a cashier abnormal behavior detection and association method. First, by analyzing the abnormal monitoring video clips, the position coordinate information of the abnormal sold items in the picture is intelligently extracted. Based on this, the system automatically generates an abnormal item marking frame and accurately draws it on the actual position of the abnormal item. It is worth mentioning that the center point of the marking frame completely coincides with the center point of the position of the abnormal item, ensuring the accuracy of the identification. Through this visual marking method, the abnormal items are clearly highlighted in the video screen, and the manager can quickly focus on the key items where the abnormal behavior occurs without viewing them frame by frame or manually identifying them. This not only greatly saves time and energy for abnormal audits, but also reduces the difficulty of identifying non-obvious abnormal behaviors. At the same time, accurate positioning of abnormal items also provides a direct basis for subsequent responsibility identification and problem tracing, which helps managers accurately analyze the causes of abnormalities and formulate corrective measures in a timely manner. Therefore, the introduction of the abnormal item marking frame greatly improves the effect and efficiency of abnormal information presentation, makes abnormal management more accurate and efficient, and provides strong support for optimizing cashier processes and standardizing cashier banking. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart of a cashier abnormal behavior detection and association method in an embodiment of the present application.

[0028] Figure 2 This is another flow chart of a cashier abnormal behavior detection and association method in an embodiment of the present application.

[0029] Figure 3 It is a schematic diagram of the physical device structure of a cashier abnormal behavior detection and association system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to any or all possible combinations comprising one or more listed items.

[0031] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0032] An application scenario of an embodiment of the present application is described below: In a large supermarket chain, the cashier was busy checking out the goods for the customers at the checkout counter. Due to the large number of customers, the cashier worked at a fast pace, and it was inevitable that some mistakes would occur. For example, when scanning the barcode of the product, an unscanned product was mistakenly put into the shopping bag; when weighing vegetables, the electronic scale was not reset to zero, causing the customer to pay more.

[0033] These seemingly minor mistakes may cause continuous economic losses to the supermarket. In order to solve this problem, the supermarket management decided to introduce an advanced abnormal behavior detection system. Based on machine vision technology, the system analyzes the surveillance video above the cashier counter to identify the cashier's operating behavior in real time and compare it with the transaction data of the POS machine to promptly detect abnormal behavior during the cashier process.

[0034] However, in actual application, the system also faces some challenges. The supermarket has a large flow of people, and customers frequently move in front of the checkout counter, blocking the camera's field of view and affecting the accuracy of abnormal behavior recognition. At the same time, different cashiers have different operating habits, and it is difficult for the system to establish a unified behavior model. In addition, for detected abnormal behaviors, the system can only prompt the existence of abnormalities, but cannot clarify the specific products and reasons for the abnormality, which brings inconvenience to subsequent problem verification and rectification.

[0035] Faced with these challenges, supermarket management realized that relying solely on video analysis or transaction data analysis can no longer meet the needs of increasingly complex checkout environments and refined management. There is an urgent need for an intelligent solution that can fully integrate video information, transaction information, and product information to achieve real-time monitoring, accurate identification, and rapid response to the entire checkout process. In order to solve the above technical problems, this application provides a cashier abnormal behavior detection and association method, which is used to improve the efficiency and accuracy of cashier abnormal behavior detection, thereby improving the efficiency of cashier loss prevention.

[0036] Combine the following Figure 1 , a cashier abnormal behavior detection and association method in an embodiment of the present application is described: See also Figure 1 , which is a flow chart of a cashier abnormal behavior detection and association method in an embodiment of the present application.

[0037] S101, receiving real-time monitoring video sent by a monitoring device and real-time order information sent by a cash register device; The system receives the real-time monitoring video sent by the monitoring device and the real-time order information sent by the cash register device. Specifically: the real-time monitoring video is subjected to denoising and contrast adjustment to obtain a pre-processed monitoring video; Use object detection algorithms to identify items for sale in pre-processed surveillance videos; Determine the sold goods containing barcodes as sold scanned goods; The sold goods without a barcode are determined as sold weighed goods.

[0038] In this step, the system receives real-time video streams from monitoring devices and real-time order information from cash registers. To ensure the quality of subsequent processing, the system first pre-processes the acquired video stream data. Through the denoising algorithm, the system can effectively eliminate noise and interference in the video and improve the clarity of the image. At the same time, through contrast adjustment, the system can enhance the layering of the video image and make key targets more prominent.

[0039] After preprocessing, the system uses advanced target detection algorithms to identify and locate the products appearing in the video. Common target detection algorithms include YOLO, SSD, Faster R-CNN, etc., which can quickly and accurately find the product targets in the video. After identifying the product, the system further determines whether there is a barcode on the product. Through barcode recognition technology, the system can automatically classify products with barcodes as sold scanned products, and products without barcodes as sold weighed products.

[0040] For example, when a customer buys a box of milk and some loose apples, the cashier places the milk and apples on the cash register. The monitoring device captures this scene and transmits the video stream to the anomaly detection system. After the system denoises and optimizes the contrast of the video, it uses the YOLO algorithm to detect that a box of milk and several apples appear in the picture. By identifying the barcode on the milk package, the system classifies the milk as a sold scanned item; while the apples do not have a barcode and are classified as sold weighed items. At the same time, the cash register also sends the order information of this transaction to the system, including the product name, quantity, price, etc. In this way, the system obtains data in two dimensions: video information and order information, which provides a basis for subsequent anomaly analysis.

[0041] S102, determining the sold scanned goods and the sold weighed goods according to the real-time monitoring video, and determining the scanning status of the sold scanned goods and the weighing status of the sold weighed goods; The system determines the sold scanned goods and the sold weighed goods based on the real-time monitoring video, and determines the scanning status of the sold scanned goods and the weighing status of the sold weighed goods. The scanning status includes scanned and unscanned, and the weighing status includes weighed and unweighed. How to define the scanning status and weighing status will be described in conjunction with another embodiment below, and will not be repeated here.

[0042] S103, determining the current settlement commodity according to the real-time order information; The system determines the current settlement product based on the real-time order information. The current settlement product includes the current settlement product information and the current time point.

[0043] In this step, the system determines the goods currently being settled, i.e. the current settlement goods, based on the real-time order information sent by the cash register device. This information usually includes the name, quantity, unit price, total price, etc. of the goods, as well as the time when the order was generated.

[0044] The system receives and analyzes the data stream from the cash register in real time and extracts key fields. By analyzing the time when the order was generated, the system can determine which transaction is currently being processed. By extracting the product information in the order, a detailed list of the current settlement products is obtained. These data provide key basis for subsequent abnormal judgment. By obtaining and analyzing order information in real time, the system can accurately grasp the transactions taking place at the cash register, laying the foundation for the identification of abnormal behavior.

[0045] S104, determining the currently processed commodity corresponding to the current time point in the real-time monitoring video; The system determines the currently processed commodity corresponding to the current time point in the real-time monitoring video, and the currently processed commodity includes currently processed commodity information.

[0046] In this step, the system needs to find the physical goods corresponding to the current settlement goods in the real-time monitoring video, that is, the current processing goods. Through video analysis technology, the system can identify the goods that the cashier is currently operating and extract its key information.

[0047] First, the system locates the corresponding time point in the video stream based on the timestamp of the order. Then, the system analyzes a short video clip before and after that time point. Through target detection and tracking algorithms, the system can automatically locate the product in the video and continuously track its movement trajectory.

[0048] Next, the system identifies and extracts attributes of the detected products. Through the pre-trained product recognition model, the system can more accurately determine the product category, brand and other information. At the same time, the system can also obtain key information on the product packaging, such as product name, specifications, production date, etc., through image segmentation, text recognition and other technologies.

[0049] For example, for the order mentioned above, the system first locates the corresponding frame in the video based on the timestamp "2023-05-01 09:30:15". Then, the system analyzes the video clips 5 seconds before and after this time point. Through target detection algorithms such as YOLO, the system finds several suspicious commodity targets in the video. After tracking these targets, it is found that one of them stayed on the checkout counter for a long time, which should be the product currently being processed.

[0050] The system zooms in and segments the product and finds that it is a box of milk. Through text recognition, the system further obtains information such as the brand name of the milk is "X Brand", the specification is "250ml", and the production date is "2023-04-15". At this point, the system has successfully extracted the key attributes of the current product "milk".

[0051] By matching the product information obtained from video analysis with the settlement products in the order, the system can associate the physical product with the virtual order. This association provides an important basis for subsequent abnormal judgment. By continuously tracking the actual operation of the cashier, the system can grasp the processing status of the product in real time and provide reliable data support for identifying abnormal behaviors such as missed scanning, missed weighing, and wrong scanning.

[0052] S105, determining the wrong scanning status according to the current settlement commodity information and the pre-processed commodity information; The system determines the wrong scanning status based on the current settlement product information and the previous processing product information. The wrong scanning status includes confirmed wrong scanning and not wrong scanning. When the current settlement product information and the previous processing product information are the same, the wrong scanning status is not wrong scanning. When the current settlement product information and the previous processing product information are different, the wrong scanning status is confirmed wrong scanning.

[0053] In this step, the system needs to determine whether the cashier has made a mistake in scanning when processing the goods. Miss-scanning means that the cashier scans and settles the goods A instead of goods B, resulting in the order information not matching the actual purchased goods. In order to identify this abnormal behavior, the system needs to compare the information of the current settlement goods and the current processing goods.

[0054] First, the system extracts the key attributes of the product, such as name, brand, and specifications, from the information of the currently processed product obtained in the previous step. Then, the system compares these attributes with the information of the currently settled product. If the two are exactly the same, it means that there is no wrong scan, and the system marks the wrong scan status as "not wrong scan". On the contrary, if there is a difference in the information of the processed product and the settled product, it is very likely that a wrong scan has occurred, and the system marks the wrong scan status as "confirmed wrong scan".

[0055] For example, let's take the previous order scenario. The system found in the video that the cashier was currently processing a box of "Brand X" milk with a specification of "250ml". The order information showed that the checkout item was also a box of milk, with the same brand and specification. In this case, the system determined that the cashier's operation was correct and marked the wrong scan status as "not wrongly scanned".

[0056] However, if the cashier picked up a box of "Y brand" milk with a specification of "200ml" in the video, and the information in the order is still "X brand" and "250ml", then the system will mark the wrong scan status as "confirmed wrong scan". This indicates that the cashier may have mistakenly scanned and settled the "Y brand" milk as "X brand" milk, which is a wrong scan behavior.

[0057] By introducing the concept of wrong scanning status, the system can accurately identify minor errors made by cashiers during operations. By comparing video information with order information in real time, wrong scanning problems can be discovered immediately to prevent customers from purchasing the wrong products or paying the wrong amount. This real-time monitoring and intelligent comparison capability is difficult to achieve with traditional manual inspections. The system can play an important role in improving cash register accuracy and reducing the risk of errors.

[0058] S106, judging whether abnormal behavior occurs according to the scanning status, weighing status and wrong scanning status; The system determines whether abnormal behavior occurs based on the scanning status, weighing status and wrong scanning status. Abnormal behavior is confirmed when one or more of the scanning status is not scanned, the weighing status is not weighed and the wrong scanning status is confirmed wrong scanning is met.

[0059] S107, determining abnormal sold items and abnormal monitoring clips corresponding to the abnormal behavior; If abnormal behavior occurs, the system will determine the abnormal sales items and abnormal monitoring clips corresponding to the abnormal behavior. After confirming that abnormal cashier behavior has occurred, the system needs to further locate the specific products and related video clips involved in the abnormal behavior, that is, abnormal sales items and abnormal monitoring clips. This step can provide support for the visualization and post-event verification of abnormal information.

[0060] First, the system will find the corresponding problematic products based on the abnormal judgment results. Taking the previous example, the system found that the apples were not weighed and the biscuits were scanned incorrectly, so the apples and biscuits were the abnormal items sold in this transaction. The system will extract the specific information of apples and biscuits from the order information, such as name, quantity, unit price, etc., to form structured abnormal product data.

[0061] Next, the system will locate the clips related to the problematic products in the video as abnormal monitoring clips. Through the previous video analysis results, the system has mastered the time points when each product appears in the video. Taking apples as an example, the system found that the cashier picked up the apples at around 09:30:20 and put the apples into the shopping bag at around 09:30:25, so this 5-second video clip is the abnormal monitoring clip of the apples. The system will record the start and end timestamps of this clip and intercept the corresponding video data from the original video stream.

[0062] In terms of technical implementation, the location of abnormally sold items can be completed through database query. The system saves the abnormal judgment results and order details in a relational database, and can easily query and extract the information of abnormal products through SQL statements. The location and interception of abnormal monitoring clips require the use of video processing technology. The system can use open source libraries such as FFmpeg to locate and edit videos according to timestamps to obtain the corresponding abnormal clip data.

[0063] S108, superimposing the real-time order information onto the corresponding abnormal monitoring segment association by an OSD character superposition method to obtain cashier abnormality information; The system uses the OSD character overlay method to overlay the real-time order information onto the corresponding abnormal monitoring clip association to obtain cashier abnormal information. After obtaining abnormal sold items and abnormal monitoring clips, the system needs to present this information in a visual way so that managers can intuitively understand the abnormal situation. A common practice is to overlay the abnormal information in the form of text, icons, etc. on the corresponding video screen through OSD (On-Screen Display) technology. This step can further improve the readability and operability of abnormal information.

[0064] Specifically, the system will first extract key information about the abnormally sold items, such as product name, abnormal type, abnormal time, etc. Then, the system will convert this information into concise and eye-catching text descriptions, such as "Product: Apple, Abnormal: Missing weight, Time: 09:30:20". This text information will be superimposed on the video screen of the abnormal monitoring clip in the form of subtitles or labels.

[0065] In terms of technical implementation, the OSD function can be implemented through FFmpeg's drawtext filter. The system first generates the configuration parameters of the drawtext filter according to the preset font, color, position and other parameters of the abnormal information. Then, the system uses the FFmpeg command to synthesize the subtitle information with the abnormal video clip to obtain a new video file. In this file, the text description of the abnormal information is superimposed on the original monitoring screen, which can intuitively show the abnormal situation to the user.

[0066] In addition to text information, the system can also superimpose some graphic elements in the video screen, such as indicator arrows, highlight boxes, etc., to further highlight the abnormal part. For example, the system can add a red box around the apple that was not weighed, and mark it with the text description "Missed product". This combination of graphic and text display can guide users to quickly locate the abnormal point and improve the readability of abnormal information.

[0067] S109, generating a viewing control, and setting the viewing control at a preset position of the cashier exception information; To further improve the usability and interactivity of abnormal information, the system will add a viewing control to the abnormal monitoring clip after it is generated. This control is similar to the "play" button on a video player. Users can easily watch the video clip related to the abnormality by simply clicking this control.

[0068] In terms of technical implementation, the system can use HTML5 <video>Tags are used to implement the video playback function. The backend first transcodes the abnormal video clip into a web-friendly format (such as MP4) and uploads it to an online storage platform (such as Alibaba Cloud OSS). Then, the front-end page uses <video>The tag introduces the URL address of the online video, and then uses CSS to set the style and position of the playback control.

[0069] S110, in response to the user clicking the view control, playing the abnormal monitoring clip; When users browse to this page, they can intuitively see the text description of the abnormal information and understand the basic situation of the abnormality. If the user wants to further view the details of the abnormality, just click the "Play" button on the player, and the system will automatically load and play the corresponding abnormal video clip. Users can preview the video content through the progress bar and focus on the abnormal part through operations such as pause and fast forward, which is very convenient and intuitive.

[0070] In addition, the system can also integrate some advanced functions on the player, such as annotation, editing, slow motion, etc. For example, when managers watch abnormal videos, they can mark and annotate key frames, or cut out key clips for archiving. These functions can further extend the application scenarios of abnormal information, allowing it to play a greater role in internal training, case analysis, etc.

[0071] S111. Marking the real-time monitoring video and the real-time order information as qualified operations; If no abnormal behavior occurs, the system will mark the real-time monitoring video and real-time order information as qualified operations.

[0072] S112, uploading the real-time monitoring video and real-time order information to a preset database.

[0073] In addition to identifying and recording abnormal behaviors, the system also needs to affirm and encourage normal cashier behaviors. On the one hand, this can help cashiers establish correct operational awareness and improve their work enthusiasm; on the other hand, recording normal behaviors can also provide data support for system optimization and prevention of abnormalities. Specifically, when the system detects that there is no abnormality in a cashier behavior, the corresponding surveillance video and order information will be marked as "qualified operation". This mark can be attached to the data record in the form of tags, fields, etc., to facilitate subsequent statistics and queries. At the same time, the system can also instantly send a congratulatory message to the cashier, such as "Congratulations on completing a standardized cashier operation!", "Thank you for your hard work, keep it up!", etc., to affirm the cashier's efforts with positive incentives.

[0074] In terms of technical implementation, the system needs to add a "normal behavior" judgment logic based on the abnormal judgment. If the scanning status, weighing status, wrong scanning status and other indicators of a cashier operation are all "normal", and the entire operation process is consistent with the standard process, the system can determine that this is a "qualified operation". In the database, the system can add an is_qualified field to the monitoring table and order table, and use the Boolean value true or 1 to mark the qualified operation record.

[0075] After identifying a qualified operation, the system will also upload the relevant surveillance video and order information to the data warehouse or cloud for subsequent data analysis and model optimization. These sample data of normal operations can be used to train anomaly detection models to improve the model's ability to recognize normal behaviors, thereby reducing the false alarm rate. At the same time, this data can also be used as a reference for cashier performance appraisals, helping managers to objectively evaluate each employee's work performance.

[0076] The above embodiments have the following beneficial effects: It can receive the surveillance video sent by the monitoring device and the order information sent by the cash register device in real time, providing a sufficient data basis for the subsequent abnormal behavior detection. Secondly, by analyzing the real-time monitoring video, it can accurately identify the scanned goods sold and the weighed goods sold, and judge their scanning status and weighing status, providing key information for detecting abnormal behavior. According to the real-time order information, the current settlement goods are determined, and compared with the current processing goods in the real-time video, it can effectively determine whether there is a wrong scanning behavior. Finally, considering the scanning status, weighing status and wrong scanning status, it can accurately determine whether there is abnormal cash register behavior. Once an abnormality is found, the system will automatically extract the order information of the abnormal sold items and the related abnormal monitoring clips, and associate the two to generate cash register abnormal information, and send it to the supervision terminal and cash register device at the same time, realizing real-time early warning and prevention of abnormal behavior. This method greatly improves the detection rate and processing efficiency of cash register abnormal behavior through multi-dimensional information fusion and intelligent analysis, effectively prevents losses in the cash register link, improves the efficiency and accuracy of cash register abnormal behavior detection, and thus improves the efficiency of cash register loss prevention.

[0077] First, the real-time surveillance video is preprocessed by denoising and contrast adjustment to improve the video quality, laying the foundation for subsequent commodity identification. Secondly, the method uses advanced target detection algorithms to quickly and accurately identify all sold commodities from the video, overcoming the limitations of human eye observation. Thirdly, the method intelligently determines whether the commodity is a sold scanned commodity or a sold weighed commodity based on whether the commodity contains a barcode, thereby realizing the automatic classification of commodity categories. This method makes full use of computer vision and artificial intelligence technology to realize the automatic identification and classification of sold commodities, which not only greatly reduces labor costs and improves inspection efficiency, but also ensures the accuracy and fairness of the identification results through objective and consistent algorithm rules, providing a reliable data reference for subsequent abnormal behavior analysis.

[0078] The OSD character overlay technology is used to overlay the order information of abnormally sold items onto the corresponding abnormal monitoring video clips in real time, so that the abnormal information is perfectly associated with the abnormal process. When the manager views the cashier abnormal information, he can clearly see the sales details of each abnormal item. At the same time, the method also provides a highly user-friendly interactive function. By setting a viewing control on the cashier abnormal information interface, when the user clicks the control, the system will automatically retrieve and play the corresponding abnormal video clip, so that the user can go directly to the abnormal scene with one click and quickly and intuitively review the full picture of the abnormal behavior. This one-click access to abnormal information presentation method greatly improves the efficiency and experience of cashier abnormal management. Managers no longer need to manually search or match abnormal information and videos, nor do they need to switch between multiple systems, thus saving a lot of time and energy. At the same time, through the video playback function, managers can repeatedly observe and carefully analyze each abnormal detail, which helps to comprehensively find the root cause of the problem and formulate targeted improvement measures.

[0079] First, by analyzing the abnormal monitoring video clips, the position coordinate information of the abnormal sold items in the picture is intelligently extracted. Based on this, the system automatically generates an abnormal item marking frame and accurately draws it at the actual location of the abnormal item. It is worth mentioning that the center point of the marking frame completely coincides with the center point of the abnormal item, ensuring the accuracy of the identification. Through this visual marking method, the abnormal items are clearly highlighted in the video screen, and managers can quickly focus on the key items where abnormal behavior occurs without viewing them frame by frame or manually identifying them. This not only greatly saves time and energy for abnormal audits, but also reduces the difficulty of identifying non-obvious abnormal behaviors. At the same time, accurate positioning of abnormal items also provides a direct basis for subsequent responsibility identification and problem tracing, which helps managers accurately analyze the causes of abnormalities and formulate corrective measures in a timely manner. Therefore, the introduction of the abnormal item marking frame greatly improves the effect and efficiency of abnormal information presentation, makes abnormal management more accurate and efficient, and provides strong support for optimizing the cashier process and standardizing cashier banking.

[0080] Intelligent analysis of abnormal monitoring clips accurately identifies the direct person responsible for each abnormal behavior, that is, the abnormal staff. Accurately linking abnormal information with specific staff makes abnormal tracing more specific and in place. On this basis, the method automatically determines whether it is necessary to send a warning reminder to the staff involved according to the preset rules. Once the warning condition is triggered, the system will push the warning information containing the details of the abnormal sold items to the cash register terminal of the corresponding staff in real time. This point-to-point abnormal reminder method ensures the accurate delivery of problem information, allowing front-line staff to realize their mistakes at the first time, so as to quickly take remedial measures to minimize the impact of the abnormality. At the same time, the method will also report the relevant information of the abnormal staff to the supervision terminal, providing data support for managers to fully control the abnormal situation and carry out subsequent assessment and accountability. By accurately locking the circulation scope of abnormal information within the scope of relevant responsible persons, it not only improves the timeliness of abnormal handling, but also avoids the abuse and interference of information release.

[0081] Through intelligent analysis of real-time video and order information, the system can automatically determine whether each cashier operation is standardized and qualified. For operations without abnormal behavior, the method will mark them as qualified and clearly distinguish between normal cashier behavior and abnormal cashier behavior. This clear behavioral distinction helps managers accurately evaluate the performance of cashiers and objectively understand and judge the overall compliance of the cashier process. At the same time, the method will also archive all real-time monitoring videos and order information and upload them to the designated database. Through standardized and systematic data storage, it is not only convenient for subsequent queries and traceability, but also lays a data foundation for in-depth applications such as big data analysis and behavior modeling. Based on complete and real historical data, managers can summarize common problems in cashier work, characterize benchmark behavior characteristics, and provide decision-making basis for optimizing training programs and improving management measures. From a higher level, through continuous accumulation and analysis of massive cashier data, it helps companies gain insight into sales patterns, optimize marketing strategies, and improve customer experience.

[0082] In conjunction with another embodiment, the step of determining the sold scanned goods and the sold weighed goods according to the real-time monitoring video in S102 of the above embodiment, and determining the scanning status of the sold scanned goods and the weighing status of the sold weighed goods will be described in detail. Figure 2 , another cashier abnormal behavior detection and association method in the embodiment of the present application is described: See also Figure 2 , which is another flow chart of a cashier abnormal behavior detection and association method in an embodiment of the present application.

[0083] S201, determining the sold scanning order information and sold weighing order information corresponding to the sold scanned goods and the sold weighed goods according to the real-time monitoring video; During the checkout process, the system needs to monitor the camera above the checkout counter in real time and identify the goods purchased by customers through video analysis technology. The system will first extract the key frames in the video stream, and then use target detection and image recognition algorithms to locate and classify the goods in the frame. For scanned goods, the system obtains the number, name, price and other information of the goods by identifying the barcode or QR code on the outer packaging of the goods; for weighed goods, the system matches the corresponding product information from the product library by identifying the appearance characteristics of the goods, such as color, shape, size, etc.

[0084] After identifying the product information, the system will match it with the order information generated in real time. For example, if the system finds that the cashier scanned a bag of potato chips and a bottle of Coke in the video, it will look for corresponding product entries in the order recorded by the system. If a matching product is found, the system will mark it as a sold scanned product and record the corresponding order number, product name, quantity, unit price and other information to form the sold scanned order information. Similarly, if the system recognizes that the cashier weighed some apples, it will look for the apple entry in the order and generate the sold weighed order information.

[0085] In terms of technical implementation, product identification can be achieved with the help of mature computer vision libraries such as OpenCV and TensorFlow. The system first uses the algorithms provided by these libraries to pre-process the video frames, such as cropping, smoothing, and enhancement, to improve image quality. Then, the system uses pre-trained target detection models (such as YOLO, SSD, etc.) to locate the products, and uses classification models (such as VGG, ResNet, etc.) to determine the category of the products. The system also needs to maintain a product feature library to record the visual features of different products, such as packaging patterns, external dimensions, etc., for matching and weighing products.

[0086] Order matching can be achieved through database association query. The system saves the product information (such as product ID, quantity, inspection time, etc.) obtained from video analysis in a temporary table, and then connects it with the order table to find order records that match the time, product, and other fields. In order to improve query efficiency, the system can create a suitable index for the order table, such as a B+ tree index based on timestamp.

[0087] S202, obtaining the processed order information entered by the cash register device; In addition to analyzing real-time surveillance video, the system also needs to obtain the processed order information entered by the cash register device, that is, the order record actually operated by the cashier. The purpose of this step is to compare the video analysis results with the internal data of the system to find possible cash register anomalies.

[0088] Cash registers generally save order information in the business database, including order number, creation time, cashier ID, product details, payment method, payment amount, etc. The system can regularly pull the latest order data from the business database through database query statements. In order to avoid frequent queries causing pressure on the business system, the system can use incremental synchronization to only obtain order records added after the last synchronization each time.

[0089] In terms of technical implementation, the system first needs to establish a connection with the business database, such as using JDBC to connect to a MySQL database. Then, the system uses SQL statements to query order data that meets the conditions, such as filtering by time range, cash register ID and other fields. In order to obtain incremental data, the system can set a synchronization mark field (such as is_synced) on the order table, or record the maximum order number of each synchronization, and then use the WHERE clause in the query condition to filter.

[0090] The order data obtained by the query is usually in JSON or XML format, containing multiple fields and nested structures. The system needs to parse and convert this data, extract key fields (such as product name, quantity, amount, etc.), and store them in the anomaly detection system's own database for subsequent comparison with the video analysis results.

[0091] S203: when it is determined that the processed order information includes the sold scan order information, determining the scanning status of the sold scanned commodity corresponding to the sold scan order information as scanned; S204: if it is determined that the processed order information does not include the sold scanned order information, the scanning status of the sold scanned commodity corresponding to the sold scanned order information is determined as unscanned; After obtaining the sold product order from the video analysis and the actual order entered into the cash register system, the system needs to compare the two to determine whether each sold product has completed the normal scanning operation. This comparison process can be divided into two situations: the scanned product exists in the actual order, and the scanned product does not exist in the actual order.

[0092] For the first case, if the system finds a record matching the scanned product sold in the actual order, it can be considered that the product was sold normally and the cashier scanned the barcode of the product during the transaction. At this time, the system marks the scanning status of this product as scanned, indicating that there is no abnormal behavior of missing scans.

[0093] Specifically, the system first traverses each product in the sold scanned order and extracts key information such as product name and quantity. Then, the system searches the processed order data to see if there is a record with the exact same product name and quantity. If a matching record is found, the system updates the scan status of the product in the video order to scanned and associates it with the actual order record, such as recording the order number, scanning time, and other information of the actual order.

[0094] For the second case, if the system cannot find a record matching the scanned product sold in the actual order, it means that the product may not have been scanned properly, and there is a risk of the cashier missing it. At this time, the system marks the scanning status of this product as unscanned, indicating that an abnormal behavior of scanning omission has occurred.

[0095] Similar to the previous example, the system iterates through the scanned items sold, and then tries to find a record with the same item name and quantity in the actual order. If no matching record is found, the system sets the item status to unscanned, and records the time when the exception occurred, the cashier, and other information for post-sales exception tracking and processing.

[0096] S205: when it is determined that the processed order information includes the sold weighing order information, the weighing status of the sold weighing commodity corresponding to the sold weighing order information is determined as weighed; In addition to scanning products, weighing products is also a common sales type in supermarkets. The system also needs to determine whether the weighed products sold have completed normal weighing operations to prevent sales losses caused by cashiers missing weights.

[0097] When processing weighed goods, the system will first compare the sold weighing order obtained by video analysis with the processed orders entered into the cash register system. If a record matching a certain product in the sold weighing order is found in the processed order, the system can assume that the product was sold normally and the weighing operation has been completed. At this time, the system marks the weighing status of the product as weighed, indicating that no abnormality has occurred.

[0098] Specifically, the system extracts the product name, weight, amount and other information for each product in the sold weighed order, and then searches the processed orders to see if there is a record with all fields consistent except the quantity. The reason why the quantity is not compared is that the quantity of weighed products in the order is usually weight, not the number of pieces. After finding a matching record, the system sets the weighing status of the product in the video order to weighed and associates it with the corresponding actual order.

[0099] S206: When it is determined that the processed order information does not include the sold weighed order information, the weighing status of the sold weighed commodity corresponding to the sold weighed order information is determined as unweighed.

[0100] Similar to scanning items, weighed items may be missed by the cashier when they are sold. If the system cannot find a record matching the weighed items sold in the processed orders, it means that these items may not be properly weighed and priced, and there is a risk of sales loss. At this time, the system will mark the weighing status of the sold items as unweighed to prompt management to pay attention to these abnormal transactions.

[0101] Specifically, the system traverses each product in the sold weighing order, first extracts key information such as product name, weight, and amount, and then tries to find records in the processed orders with the same fields except quantity. If no records that meet the conditions are found, the system determines that the product is likely not weighed by the cashier, marks its weighing status as unweighed, and records the time when the abnormality occurred, the cashier, the product, and other information, providing clues for subsequent problem tracking and rectification.

[0102] The above embodiments have the following beneficial effects: By using the real-time monitoring video and the order information of the cash register equipment, the real processing status of each sold commodity can be determined efficiently and accurately through intelligent matching and comparison of the two. First, the order information corresponding to the sold scanned goods and weighed goods is extracted from the real-time video, and then the processed orders entered by the cash register equipment are obtained. By comparing the two types of order information, it can be determined whether the sold goods identified in the video have been scanned or weighed in the cash register system. This audit method combining video and order, on the one hand, makes up for the shortcomings of a single data dimension and realizes multi-angle and all-round status determination; on the other hand, it avoids the subjectivity and differences of manual verification and ensures the consistency and authority of the determination results. Through automated and precise status determination, this method can detect omissions or erroneous behaviors in the cash register process in the first time, providing a basis for abnormal warning and problem rectification, which is conducive to improving cash register efficiency, standardizing cash register processes, and reducing operating risks. It has a positive effect on maintaining the interests of shopping malls and improving customer satisfaction.

[0103] The following describes the system in the embodiment of the present invention from the perspective of hardware processing. Figure 3 , which is a schematic diagram of the physical device structure of a cashier abnormal behavior detection and association system provided in an embodiment of the present application.

[0104] It should be noted that Figure 3 The structure of the system shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0105] like Figure 3 As shown, the system includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 302 or the program loaded from the storage part 308 to the random access memory (RAM) 303, such as executing the method in the above embodiment. In RAM 303, various programs and data required for system operation are also stored. CPU 301, ROM 302 and RAM 303 are connected to each other through a bus 304. Input / output (I / O) interface 305 is also connected to bus 304.

[0106] The following components are connected to the I / O interface 305: an input section 306 including a camera, an infrared sensor, etc.; an output section 307 including a liquid crystal display (LCD) and a speaker, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 310 as needed so that a computer program read therefrom is installed into the storage section 308 as needed.

[0107] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 309, and / or installed from a removable medium 311. When the computer program is executed by the central processing unit (CPU) 301, various functions defined in the present invention are performed.

[0108] It should be noted that the computer-readable medium shown in the embodiment of the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable computer program. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing.

[0109] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. Among them, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0110] As another aspect, the present invention further provides a computer-readable storage medium, which may be included in the system described in the above embodiment; or may exist independently without being assembled into the system. The above storage medium carries one or more computer programs, and when the above one or more computer programs are executed by a processor of a system, the system implements the method provided in the above embodiment.

[0111] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0112] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.

[0113] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0114] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.< / video> < / video>

Claims

1. A cashier abnormal behavior detection and association method, characterized in that: include: Receive real-time surveillance video sent by monitoring equipment and real-time order information sent by cash register equipment; Determine the sold scanned goods and the sold weighed goods according to the real-time monitoring video, and determine the scanning status of the sold scanned goods and the weighing status of the sold weighed goods, wherein the scanning status includes scanned and unscanned, and the weighing status includes weighed and unweighed; Determine the current settlement commodity according to the real-time order information, wherein the current settlement commodity includes the current settlement commodity information and the current time point; Determining a currently processed commodity corresponding to the current time point in the real-time monitoring video, wherein the currently processed commodity includes currently processed commodity information; Determine the wrong scanning status according to the current settlement commodity information and the pre-processed commodity information, the wrong scanning status including confirmed wrong scanning and not wrong scanning, when the current settlement commodity information and the pre-processed commodity information are the same, the wrong scanning status is not wrong scanning, when the current settlement commodity information and the pre-processed commodity information are different, the wrong scanning status is confirmed wrong scanning; Determine whether abnormal behavior occurs according to the scanning state, the weighing state and the wrong scanning state, and confirm that the abnormal behavior occurs when one or more of the scanning state is not scanned, the weighing state is not weighed and the wrong scanning state is confirmed as wrong scanning; If the abnormal behavior occurs, determining the abnormal sold items and abnormal monitoring clips corresponding to the abnormal behavior; The real-time order information of the abnormally sold items is associated with the abnormal monitoring segment to obtain cashier abnormality information, and the cashier abnormality information is sent to the supervision terminal and the cashier device.

2. The method according to claim 1, characterized in that Determining the sold scanned goods and the sold weighed goods according to the real-time monitoring video specifically includes: Performing denoising and contrast adjustment on the real-time monitoring video to obtain a pre-processed monitoring video; identifying sold goods in the preprocessed surveillance video using an object detection algorithm; Determining the sold commodities containing barcodes as the sold scanned commodities; The sold commodities that do not contain the barcode are determined as the sold weighed commodities.

3. The method according to claim 1, characterized in that The determining of the scanning status of the sold scanned commodity and the weighing status of the sold weighed commodity specifically includes: Determining the sold scanning order information and the sold weighing order information corresponding to the sold scanning goods and the sold weighing goods according to the real-time monitoring video; Obtaining processed order information entered by the cash register device; When it is determined that the processed order information includes the sold scan order information, determining the scanning status of the sold scan commodity corresponding to the sold scan order information as scanned; When it is determined that the processed order information does not include the sold scanned order information, determining the scanning status of the sold scanned commodity corresponding to the sold scanned order information as unscanned; In the case where it is determined that the processed order information includes the sold weighed order information, determining the weighing status of the sold weighed commodity corresponding to the sold weighed order information as weighed; When it is determined that the processed order information does not include the sold weighed order information, the weighing status of the sold weighed commodity corresponding to the sold weighed order information is determined as unweighed.

4. The method according to claim 1, characterized in that The step of associating the real-time order information of the abnormally sold items with the abnormal monitoring segment to obtain the cashier abnormality information specifically includes: The real-time order information is superimposed on the corresponding abnormal monitoring segment association by an OSD character superposition method to obtain the cashier abnormality information; Generate a viewing control, and set the viewing control at a preset position of the cashier exception information; In response to the user clicking the viewing control, the abnormal monitoring segment is played.

5. The method according to claim 4, characterized in that After generating a viewing control and setting the viewing control at a preset position of the cashier exception information, the method further includes: Determine the location information of the abnormal sold item according to the abnormal monitoring segment; An abnormal object marking frame is generated according to the position information, and a center point of the abnormal object marking frame is the same as a center point of the position information.

6. The method according to claim 1, characterized in that After associating the real-time order information of the abnormally sold items with the abnormal monitoring segment to obtain abnormal cashier information, and sending the abnormal cashier information to the supervision terminal and the cashier device, the method further includes: Determining an abnormal staff member who handles the abnormal sold item in the abnormal monitoring segment; Determine whether to send a warning message to the cashier terminal corresponding to the abnormal staff member according to a preset setting, wherein the warning message includes the real-time order information of the abnormally sold item; Obtain relevant information of the abnormal staff member and send the relevant information to the supervision terminal.

7. The method according to claim 1, characterized in that After associating the real-time order information of the abnormally sold items with the abnormal monitoring segment to obtain abnormal cashier information, and sending the abnormal cashier information to the supervision terminal and the cashier device, the method further includes: If the abnormal behavior does not occur, marking the real-time monitoring video and the real-time order information as qualified operations; The real-time monitoring video and the real-time order information are uploaded to a preset database.

8. A cashier abnormal behavior detection and association system, characterized in that: The system comprises: One or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the system to execute the method as described in any one of claims 1-7.

9. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on a system, the system is caused to execute the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that When the computer program product is run on a system, the system is caused to execute the method according to any one of claims 1 to 7.

Images