A method for detecting a violation of a customer operation and related devices

Abstract

The application discloses a method for detecting illegal customer operation and related devices, which can be applied to the field of artificial intelligence or the field of finance. Video stream data collected by a camera is obtained, target recognition operation is performed on the video stream data, and personnel detection results and article detection results are obtained. If the personnel hand position of the first type of personnel in the personnel detection results is empty, prediction operation is performed on the personnel hand position of the first type of personnel, and the three-dimensional coordinates of the personnel hand position of the first type of personnel are determined according to the personnel hand position of the first type of personnel corresponding to each video stream data. In addition, the three-dimensional coordinates of the target device are determined. That is, through the above steps, the three-dimensional coordinates of the hand of the staff and the three-dimensional coordinates of the customer's mobile phone are detected, and when the relative distance between the three-dimensional coordinates of the hand of the staff and the three-dimensional coordinates of the customer's mobile phone is not in compliance, illegal customer operation warning information is determined, and the detection of the mobile phone of the customer operation is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of finance, and in particular to a method for detecting illegal customer operation and related device. BACKGROUND

[0002] Customer operation mobile phone refers to the phenomenon that bank staff use customers' mobile phones to complete some specific operations on behalf of customers when customers need help or for the purpose of convenience. This behavior is illegal at bank outlets.

[0003] Customer operation mobile phone may cause many risks and hazards, such as customer information leakage, staff stealing customer funds, and violation of laws and regulations. To avoid this illegal operation, the conventional inspection method is manual patrol and monitoring video spot check, but these methods have low detection accuracy. SUMMARY

[0004] In view of the above problems, the present application provides a method for detecting illegal customer operation and related device to improve the detection accuracy of customer operation mobile phone. The specific scheme is as follows:

[0005] The first aspect of the present application provides a method for detecting illegal customer operation, comprising:

[0006] Obtaining video stream data collected by multiple camera devices;

[0007] Performing target recognition operation on the video stream data to obtain personnel detection results and article detection results; the personnel detection results include personnel categories and human body key point positions of each personnel; the article detection results include types and positions of articles;

[0008] If the personnel hand position in the human body key point position of the first type of personnel in the personnel detection results is empty, predicting the personnel hand position of the first type of personnel in the video stream data by using the personnel detection results corresponding to the historical video stream;

[0009] Determining three-dimensional coordinates of the personnel hand position of the first type of personnel according to the personnel hand position of the first type of personnel corresponding to each video stream data;

[0010] Analyzing the trajectory of the target device of the second type of personnel based on the article detection results corresponding to each video stream data to obtain two-dimensional coordinates of the target device in the video stream data;

[0011] Determining three-dimensional coordinates of the target device according to the two-dimensional coordinates of the target device in the video stream data;

[0012] Based on the relative distance between the three-dimensional coordinates of the hand position of the first type of personnel and the three-dimensional coordinates of the target device, a warning message for unauthorized valet operation is determined.

[0013] In one possible implementation, using the personnel detection results corresponding to historical video streams, the position of the hands of the first type of personnel in the video stream data is predicted, including:

[0014] From the human body key point locations in the personnel detection results corresponding to the historical video stream, the locations of each key point of the first type of personnel are selected;

[0015] Based on the logical relationship between non-hand key point locations and hand key point locations, the hand locations of the first type of personnel are predicted in the video stream data by utilizing the locations of various key points of the first type of personnel.

[0016] In one possible implementation, determining the three-dimensional coordinates of the hand position of the first type of person based on the hand position of the person corresponding to each of the video stream data includes:

[0017] Calculate the parallax data of the hand position of the first type of person in different video stream data;

[0018] Based on the parallax data, calculate the relative distance between the hands of the first type of person and the corresponding camera device;

[0019] Using the relative distance, the three-dimensional coordinates of the hand position of the first type of person are calculated.

[0020] In one possible implementation, based on the item detection results corresponding to each of the video stream data, the trajectory of the target device for the second type of person is analyzed to obtain the two-dimensional coordinates of the target device in the video stream data, including:

[0021] The target equipment of the second type of personnel will be used as the target to be tracked;

[0022] Based on the corresponding item detection results, determine the appearance features of the items in the video stream data;

[0023] Extract the appearance features of the target to be tracked;

[0024] Based on the similarity between the appearance features of the item and the appearance features of the target to be tracked, the trajectory of the target to be tracked is determined to obtain the two-dimensional coordinates of the target device in the video stream data.

[0025] In one possible implementation, the target device of the second type of personnel is taken as the target to be tracked, including:

[0026] screening a second type of personnel from the personnel detection result corresponding to the video stream data;

[0027] taking a device closest to the second type of personnel when the device first appears as a target device;

[0028] taking the target device as a tracking target.

[0029] In a possible implementation, the rule-violating customer operation early warning information is determined based on a relative distance between a three-dimensional coordinate of a hand position of the first type of personnel and a three-dimensional coordinate of the target device, and includes:

[0030] calculating a Euclidean distance between the three-dimensional coordinate of the hand position of the first type of personnel and the three-dimensional coordinate of the target device;

[0031] if the relative distance is less than a preset threshold for a time greater than a preset time and there is a behavior of the first type of personnel operating the target device, determining that the rule-violating customer operation early warning information is an identifier representing a rule violation.

[0032] The second aspect of the application provides a rule-violating customer operation detection device, including:

[0033] a data acquisition module configured to acquire video stream data collected by a plurality of camera devices;

[0034] a target recognition module configured to perform target recognition on the video stream data to obtain personnel detection results and article detection results; the personnel detection results include personnel categories and body key point positions of each personnel; and the article detection results include types and positions of articles;

[0035] a position prediction module configured to, if a hand position of a first type of personnel in the body key point positions in the personnel detection results is empty, predict the hand position of the first type of personnel in the video stream data by using personnel detection results corresponding to historical video stream data;

[0036] a first coordinate determination module configured to determine a three-dimensional coordinate of the hand position of the first type of personnel according to the hand position of the first type of personnel corresponding to each of the video stream data;

[0037] a second coordinate determination module configured to analyze a trajectory of a target device of a second type of personnel based on the article detection results corresponding to each of the video stream data to obtain a two-dimensional coordinate of the target device in the video stream data;

[0038] a third coordinate determination module configured to determine a three-dimensional coordinate of the target device according to the two-dimensional coordinate of the target device in the video stream data.

[0039] a rule violation detection module configured to determine a rule violation warning information based on a relative distance between a three-dimensional coordinate of a hand position of the first type of personnel and a three-dimensional coordinate of the target device.

[0040] A third aspect of the present application provides a computer program product, comprising computer readable instructions which, when executed on an electronic device, cause the electronic device to implement the rule violation detection method described above.

[0041] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0042] the memory is configured to store a computer program;

[0043] the processor is configured to execute the computer program to enable the electronic device to implement the rule violation detection method described above.

[0044] A fifth aspect of the present application provides a computer storage medium, the storage medium carrying one or more computer programs, when the one or more computer programs are executed by an electronic device, the electronic device can implement the rule violation detection method described above.

[0045] By employing the above technical solution, this application provides a method and related apparatus for detecting unauthorized proxy operations. In this application, video stream data collected by multiple camera devices is acquired, and target recognition is performed on the video stream data to obtain personnel detection results and item detection results. The personnel detection results include the personnel category and key point positions of each person; the item detection results include the type and location of the item. If, in the personnel detection results, the hand position of a first type of person is empty in the key point positions, the hand position of the first type of person in the video stream data is predicted using the personnel detection results corresponding to historical video streams. Based on the hand positions of the first type of person corresponding to each video stream data, the three-dimensional coordinates of the hand position of the first type of person are determined. Furthermore, based on the item detection results corresponding to each video stream data, the trajectory of the target device for a second type of person is analyzed to obtain the two-dimensional coordinates of the target device in the video stream data. Based on the two-dimensional coordinates of the target device in the video stream data, the three-dimensional coordinates of the target device are determined. In other words, this invention, through the aforementioned steps, detects the three-dimensional coordinates of the staff member's hand and the customer's mobile phone. When the relative distance between the three-dimensional coordinates of the staff member's hand and the customer's mobile phone is non-compliant, it can identify and issue a warning for unauthorized phone operation, thus achieving the detection of unauthorized phone operation. Furthermore, the entire process is analyzed in real-time through video streams, requiring no manual intervention and saving manpower. Additionally, in this invention, if the hand position of a first-type person is not detected in the personnel detection results, the hand position of that first-type person in the video stream data is predicted using the personnel detection results corresponding to historical video streams. This allows for prediction of the staff member's hand position even when it is obscured, ensuring real-time acquisition of the staff member's hand position and improving the reliability and accuracy of detecting unauthorized phone operation. Furthermore, in this invention, based on the item detection results corresponding to each video stream data, the trajectory of the target device of the second type of person is analyzed to obtain the two-dimensional coordinates of the target device in the video stream data. Based on the two-dimensional coordinates of the target device in the video stream data, the three-dimensional coordinates of the target device are determined. This enables real-time monitoring of the user's mobile phone movement trajectory, preventing the target from being lost during the handover process, and also improving the accuracy and reliability of mobile phone detection for surrogate operation. Attached Figure Description

[0046] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0047] Figure 1A system architecture schematic diagram provided for the present application;

[0048] Figure 2 A terminal structure schematic diagram provided for the present application;

[0049] Figure 3 An edge computing device structure schematic diagram provided for the present application;

[0050] Figure 4 A flow chart of a method for detecting illegal customer operation provided for the present application;

[0051] Figure 5 A flow chart of a method for determining coordinates provided for the present application;

[0052] Figure 6 A structure schematic diagram of a device for detecting illegal customer operation provided for the present application;

[0053] Figure 7 An electronic device structure schematic diagram provided for the present application. DETAILED DESCRIPTION

[0054] The embodiments of the present application are described below in conjunction with the accompanying drawings. The terms used in the embodiment part of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0055] The embodiments of the present application are described below in conjunction with the accompanying drawings. It is known to those of ordinary skill in the art that, as technology develops and new scenarios appear, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0056] The terms "first", "second", and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, and this is only a distinguishing way used in the description of the embodiments of the present application to describe the objects with the same attributes. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, so that the processes, methods, systems, products or devices containing a series of units do not necessarily limit to those units, but can include other units not clearly listed or inherent to these processes, methods, products or devices.

[0057] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0058] The mobile phone operation by the agent refers to the phenomenon that the bank staff replaces the customer to use the customer's mobile phone to complete some specific operations when the customer needs help or for the purpose of convenience, which is a violation in the bank outlet.

[0059] The mobile phone operation by the agent will bring many risks and hidden dangers, such as customer information leakage, staff stealing customer funds, violation of laws and regulations, etc. In order to avoid this violation, the conventional inspection method is manual patrol and monitoring video spot check, but these methods are easy to be affected by human subjective discretion, and the detection accuracy is low. In addition, there are problems of high labor cost and low inspection efficiency.

[0060] In addition, there is a method of judging the category of personnel through a face matching algorithm, and then judging whether there is a violation of the operation by the agent. However, in the bank scene, the camera angle of the mobile phone operation scene needs to be able to clearly see the mobile phone, so the angle of the camera is mostly vertical, and the frontal information of the person cannot be obtained, so this method of judging the identity of the person is not applicable in the scene of the mobile phone operation by the agent.

[0061] In view of the problems of the existing mobile phone operation by the agent violation behavior inspection method, the embodiment of the present application automatically detects the violation of the mobile phone operation by the agent by using a visual model and an edge computing algorithm, which can check the operation of the mobile phone in real time and timely find abnormal behavior and violation operation. The present application is more suitable for the actual bank scene, greatly reduces the possibility of manual intervention and misjudgment, and can also count and analyze the detection data, providing decision support and supervision basis for the bank management.

[0062] Specifically, the visual model and the edge computing algorithm are deployed in the edge computing box, and the function of real-time automatic early warning of the violation behavior is realized through the management platform. The personnel information is obtained by using the visual model, including the category of personnel, personnel action recognition, and whether the staff operates the mobile phone by the agent is judged by combining the personnel information through the personnel behavior analysis algorithm. When the violation behavior occurs, the violation information is pushed to the management platform in time, the management platform records and counts the violation behavior, and the relevant personnel are notified in real time to handle the early warning information, realizing the visual automatic inspection of the violation behavior.

[0063] In detail, the application provides a method for detecting illegal customer operation and related device. In the application, video stream data collected by multiple camera devices is obtained, target recognition operation is performed on the video stream data, and personnel detection results and article detection results are obtained. The personnel detection results include personnel categories and human body key point positions of each personnel. The article detection results include types and positions of articles. If the personnel hand position in the human body key point position of the first type personnel in the personnel detection results is empty, the personnel hand position of the first type personnel in the video stream data is predicted by using the personnel detection results corresponding to historical video stream data, and the three-dimensional coordinates of the personnel hand position of the first type personnel are determined according to the personnel hand positions of the first type personnel corresponding to each video stream data. In addition, the trajectory of a target device of a second type personnel is analyzed based on the article detection results corresponding to each video stream data, so as to obtain the two-dimensional coordinates of the target device in the video stream data, and the three-dimensional coordinates of the target device are determined according to the two-dimensional coordinates of the target device in the video stream data. That is, by the above steps, the three-dimensional coordinates of the hands of the staff and the three-dimensional coordinates of the customer's mobile phone are detected, the illegal customer operation warning information can be determined when the relative distance between the three-dimensional coordinates of the hands of the staff and the three-dimensional coordinates of the customer's mobile phone is not in compliance, the detection of the customer's operation mobile phone is realized, and the whole process is analyzed in real time through the video stream, without the need for manual participation, thereby saving manpower. In addition, in the application, when the personnel hand position of the first type personnel is not detected in the personnel detection results, the personnel hand position of the first type personnel in the video stream data is predicted by using the personnel detection results corresponding to the historical video stream data, so that the hand position of the staff can be predicted when the hand of the staff is blocked, the hand position of the staff can be obtained in real time, and the reliability and accuracy of the detection of the customer's operation mobile phone are improved. In addition, in the application, the trajectory of the target device of the second type personnel is analyzed based on the article detection results corresponding to each video stream data, so as to obtain the two-dimensional coordinates of the target device in the video stream data, and the three-dimensional coordinates of the target device are determined according to the two-dimensional coordinates of the target device in the video stream data, so that the motion trajectory of the user's mobile phone can be monitored in real time, the target can be avoided from being lost in the process of handover, and the accuracy and reliability of the detection of the customer's operation mobile phone can be improved.

[0064] It should be noted that the method for detecting illegal customer operation and related device provided by the application can be used in the field of artificial intelligence or the field of finance. The above is only an example and does not limit the application of the method for detecting illegal customer operation and related device provided by the application.

[0065] Reference Figure 1 , Figure 1A schematic diagram of a system architecture is shown. The system may include a terminal 100 and an edge computing device 200. The edge computing device 200 may include one or more edge computing devices (…). Figure 1 (The example includes an edge computing device 200, which can provide the methods provided in the embodiments of this application to one or more terminals.)

[0066] The terminal 100 may be equipped with a video stream acquisition application. The application and webpage can provide an interface. The terminal 100 can receive relevant parameters input by the user on the video stream acquisition interface and send the acquired video stream to the edge computing device 200. The edge computing device 200 can obtain the processing result based on the received data.

[0067] The following description Figure 1 The product form of the mid-terminal 100;

[0068] The terminal 100 in this application embodiment can be a camera device, mobile phone, tablet computer, wearable device, vehicle device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc., and this application embodiment does not impose any restrictions on it.

[0069] Figure 2 A schematic diagram of an optional hardware structure for terminal 100 is shown.

[0070] refer to Figure 2 As shown, the terminal 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), a headphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190, and other components. Those skilled in the art will understand that... Figure 2 These are merely examples of terminals or multi-functional devices and do not constitute a limitation on terminals or multi-functional devices. They may include more or fewer components than shown in the illustration, or combine certain components, or use different components.

[0071] The input unit 130 can be used to receive inputted digital or character information, and to generate key signal inputs related to user settings of the portable multifunctional device and control of functions. Specifically, the input unit 130 can include a touch screen 131 (optional) and / or other input devices 132. The touch screen 131 can collect touch operations of a user thereon or therearound (such as operations of the user using a finger, a joint, a stylus, or any suitable object on or near the touch screen), and drive corresponding connected devices according to pre-set programs. The touch screen can detect touch actions of the user on the touch screen, convert the touch actions into touch signals and send the touch signals to the processor 170, and can receive commands from the processor 170 and execute the commands; the touch signals at least include touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, the touch screen can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, the other input devices 132 can include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, on-off keys, etc.), trackballs, mice, joysticks, etc.

[0072] The input device 132 can receive inputted data, etc.

[0073] The display unit 140 can be used to display information inputted by the user or provided to the user, various menus of the terminal 100, interactive interfaces, file display, and / or playing of any kind of multimedia files. In the embodiments of the present application, the display unit 140 can be used to display interfaces of video stream collection, processing results, etc.

[0074] The storage 120 can be used to store instructions and data. The storage 120 can mainly include a storage instruction area and a storage data area. The storage data area can store various data such as multimedia files, texts, etc.; the storage instruction area can store software units such as operating systems, applications, instructions required by at least one function, etc., or their subsets, expanded sets. It can also include a non-volatile random access memory; provide the processor 170 with software and applications that include management of hardware, software, and data resources in the computing processing device, support control. It is also used for storage of multimedia files, and storage of running programs and applications.

[0075] The processor 170 is the control center of the terminal 100, connects each part of the whole terminal 100 by various interfaces and lines, executes various functions of the terminal 100 and processes data by running or executing the instructions stored in the memory 120 and calling the data stored in the memory 120, thereby performing overall control on the terminal device. Optionally, the processor 170 can include one or more processing units; preferably, the processor 170 can integrate an application processor and a modem processor, wherein the application processor mainly processes operating systems, user interfaces and application programs and the like, and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 170. In some embodiments, the processor, the memory, can be realized on a single chip, and in some embodiments, they can also be realized on independent chips respectively. The processor 170 can also be used to generate corresponding operation control signals to the corresponding components of the computing processing device, read and process the data in the software, especially read and process the data and programs in the memory 120, so that each functional module in it performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0076] The memory 120 can be used to store software codes related to the video stream acquisition method, and the processor 170 can execute the steps of the video stream acquisition method, or can also schedule other units (such as the above-mentioned input unit 130 and display unit 140) to realize corresponding functions.

[0077] The radio frequency unit 110 (optional) can be used for receiving and sending signals in the process of receiving information or calling, for example, receiving the downlink information of the base station and processing it by the processor 170; in addition, sending the uplink data designed to the base station. Usually, the RF circuit includes but is not limited to antenna, at least one amplifier, transceiver, coupler, low noise amplifier (LNA), duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to global system for mobile communication (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), long term evolution (LTE), email, short message service (SMS), etc.

[0078] In the embodiments of the present application, the radio frequency unit 110 can send data to the edge computing device 200 and receive the processing result sent by the edge computing device 200.

[0079] It should be understood that the radio frequency unit 110 is optional, which can be replaced by other communication interfaces, for example, can be a network interface.

[0080] The terminal 100 further includes a power supply 190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so that the power management system can realize the functions of managing charging, discharging, and power consumption management, etc.

[0081] The terminal 100 further includes an external interface 180, which can be a standard Micro USB interface, or a multi-pin connector, and can be used to connect the terminal 100 with other devices for communication, or can be used to connect a charger to charge the terminal 100.

[0082] Although not shown, the terminal 100 can further include a flash, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which will not be described here. Part or all of the methods described below can be applied in the terminal 100 as shown. Figure 2

[0083] Next, the product form of the edge computing device 200 is described. Figure 1

[0084] Figure 3 A structural schematic diagram of the edge computing device 200 is provided, as shown in the figure. The edge computing device 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate through the bus 201. Figure 3

[0085] The bus 201 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 only one thick line is used, but it does not mean that there is only one bus or only one type of bus.

[0086] ​​​The processor 202 can be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), etc.

[0087] The memory 204 can include volatile memory, such as random access memory (RAM), and non-volatile memory, such as read-only memory (ROM), flash memory, a hard disk drive (HDD), or a solid-state drive (SSD).

[0088] The memory 204 can be configured to store software codes related to the method for detecting illegal customer operation, and the processor 202 can execute the steps of the method for detecting illegal customer operation of the chip, or can schedule other units to implement corresponding functions.

[0089] It should be understood that the terminal 100 and the edge computing device 200 described above can be centralized or distributed devices, and the processors (such as the processor 170 and the processor 202) in the terminal 100 and the edge computing device 200 can be hardware circuits (such as an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, etc.), or a combination of these hardware circuits, for example, the processor can be a hardware system with an execution instruction function, such as a CPU, a DSP, etc., or a hardware system without an execution instruction function, such as an ASIC, an FPGA, etc., or a combination of the hardware system without an execution instruction function and the hardware system with an execution instruction function.

[0090] To solve the above problems, the embodiment of the present application provides a method for detecting illegal customer operation. The method for detecting illegal customer operation of the embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0091] Referring to Figure 4 ,Figure 4 The flowchart of a method for detecting illegal customer operation provided by an embodiment of the present application is shown in Figure 4 The method for detecting illegal customer operation provided by an embodiment of the present application can include the following steps.

[0092] S11, acquiring video stream data collected by multiple camera devices.

[0093] In actual scenarios, since the area of a bank working area is large and the monitoring range of each camera device is limited, multiple camera devices are needed to monitor the bank working area. Generally, the camera device can be a camera installed on the roof with a vertical angle.

[0094] Each camera device can collect video stream data in real time and output the collected video stream data to an edge computing device to enable the edge computing device to detect illegal customer operation.

[0095] S12, performing target recognition on the video stream data to obtain personnel detection results and article detection results; the personnel detection results include the personnel category and the human body key point position of each personnel; and the article detection results include the type and position of the article.

[0096] Specifically, in actual scenarios, illegal customer operation generally refers to that a staff member operates a customer's device such as a mobile phone or a tablet on behalf of the customer. Therefore, it is necessary to recognize the staff member, the customer, and the article in the video stream data.

[0097] Taking a mobile phone as an example of the device to be recognized, data when operating the mobile phone is acquired, public data is integrated as a training set, a target recognition model is labeled as a staff member, a customer, and a mobile phone, data is labeled, and the target recognition model is trained.

[0098] After the model training is completed, the model pt file is installed into the edge computing device, the video stream data is input into the model during model use, the recognized target is boxed with a labeled box such as a rectangular box, and a pt file is obtained, which includes the recognized target such as a staff member, a customer, and an article (such as a mobile phone).

[0099] In addition to the above-mentioned staff member, customer, and article (such as a mobile phone), a mobile phone operation behavior is also needed during customer operation. Therefore, in addition to recognizing the staff member, customer, and article (such as a mobile phone), the operation behavior needs to be detected in the embodiment of the present application. The operation behavior is the current action of the personnel.

[0100] In the detection of specific actions, a personnel action analysis model can be used, which is different from other models that focus on all 17 key points of the human body. In this embodiment, the camera angle is vertical and mainly focuses on hand actions, so the personnel action analysis model mainly focuses on the 16 key points of the upper body: shoulder, elbow, wrist, thumb tip, index tip, middle tip, ring tip, and little tip. A 16-person key point model is trained (specifically: shoulder, elbow, wrist, and the top of each finger), which can accurately determine the position of the target person's hand and the action of the target person's hand when the person is blocked or overlapped, and can effectively analyze the hand action of the person.

[0101] Therefore, the target recognition model and the personnel action analysis model are used to process the video stream data to obtain personnel detection results and article detection results.

[0102] The personnel detection results include the personnel categories and the human key point positions of each personnel, such as personnel being a staff member, the staff member's label box marking the positions of the staff member's shoulder, elbow, wrist, and the top of each finger, such as each key point being marked out with a point. The same processing is performed for the customer.

[0103] The article detection results include the types and positions of articles, such as an article being a mobile phone, and the position of the mobile phone being boxed out using a label box.

[0104] S13, if the personnel hand position in the human key point position of the first type of personnel in the personnel detection result is empty, the personnel hand position of the first type of personnel in the video stream data is predicted using the personnel detection result corresponding to the historical video stream.

[0105] In actual applications, for each video stream data collected by a camera device, some video stream data may be blocked by objects such as machines and columns, resulting in no detection of the hand of the personnel. At this time, the hand position of the corresponding personnel can be predicted by prediction.

[0106] In the case of illegal customer operation, the staff member usually operates the user's mobile phone, so it is only necessary to focus on the hand action of the staff member. Therefore, for the personnel detection result corresponding to each video stream data, if the personnel hand position in the human key point position of the first type of personnel, specifically the staff member, is empty, i.e., the staff member's hand is not detected, the position of the shoulder, elbow, and wrist in the personnel detection result of the video stream in which the hand is detected in the historical video stream collected by the same camera device can be used to predict the hand position.

[0107] If the hand position of the staff is detected in some video stream data, the prediction operation is not needed, that is, the hand position of the staff is predicted only in the video stream data in which the hand position of the staff is not detected, and finally the hand position of the staff corresponding to each video stream data is obtained.

[0108] S14, determining the three-dimensional coordinates of the hand position of the staff of the first type of staff according to the hand position of the staff of the first type of staff corresponding to each video stream data.

[0109] Specifically, the hand position of the staff of the first type of staff is a two-dimensional coordinate in the video stream data, which needs to be converted to a three-dimensional space to obtain the three-dimensional coordinates of the hand position of the staff, that is, the specific position of the hand position of the staff in the three-dimensional space.

[0110] Compared with the point cloud data, the two-dimensional coordinates and the three-dimensional coordinates in the embodiment need to occupy a smaller space and can be completely deployed in the edge computing box.

[0111] S15, analyzing the trajectory of the target device of the second type of staff based on the object detection result corresponding to each video stream data to obtain the two-dimensional coordinates of the target device in the video stream data.

[0112] After obtaining the hand position of the staff through the above steps, the actual position of the customer's mobile phone needs to be obtained to detect the illegal operation of the customer.

[0113] When detecting the actual position of the mobile phone, the position of the customer's mobile phone may be adjusted, such as being transferred from one customer's hand to another customer's hand, or being transferred to the hand of the staff. Therefore, in order to track the customer's mobile phone, the trajectory of the target device (such as the mobile phone) of the second type of staff, that is, the customer, needs to be analyzed to obtain the two-dimensional coordinates of the target device in the video stream data.

[0114] S16, determining the three-dimensional coordinates of the target device according to the two-dimensional coordinates of the target device in the video stream data.

[0115] Like the hand of the staff, the two-dimensional coordinates of the customer's mobile phone also need to be converted to obtain the three-dimensional coordinates, that is, the actual position of the customer's mobile phone in the three-dimensional space. The coordinate conversion process is the same as that of the staff, and only the processing object is different.

[0116] S17, determining the illegal operation warning information based on the relative distance between the three-dimensional coordinates of the hand position of the staff of the first type of staff and the three-dimensional coordinates of the target device.

[0117] In this embodiment, after obtaining the three-dimensional coordinates corresponding to the hand position of the staff and the three-dimensional coordinates corresponding to the position of the customer's mobile phone, the spatial distance between them can be calculated by the Euclidean distance formula. That is, the Euclidean distance of the three-dimensional coordinates of the hand position of the staff of the first type and the three-dimensional coordinates of the target device is calculated.

[0118] If the relative distance is less than the preset threshold for more than a preset time and there is a behavior of the first type of staff operating the target device, the rule-breaking customer operation warning information is determined as an identifier representing a rule-breaking. Specifically, after obtaining the distance between the staff's fingers and the customer's mobile phone, when the distance is less than a certain threshold for a certain time and there is a clear action of the staff operating the customer's mobile phone, it is determined that the staff's operation of the customer's mobile phone is rule-breaking. The action of clearly operating the mobile phone can be determined in combination with whether the background receives an instruction of the customer to handle the business and the similarity of the front and back frames of the camera monitoring the mobile phone screen is less than a certain threshold.

[0119] If it is determined that the staff's operation of the customer's mobile phone is rule-breaking, the rule-breaking customer operation warning information is determined as an identifier representing a rule-breaking. The identifier can be a rule-breaking word or a digital identifier, such as 1. The edge computing device records the rule-breaking customer operation warning information and pushes the warning information to the corresponding manager.

[0120] In this embodiment, by detecting the time when the distance is less than the threshold and whether there is an action of the staff operating the customer's mobile phone, it can be accurately determined whether there is a rule-breaking operation behavior, avoiding false positives due to the staff's large working amplitude at a certain moment, such as the staff touching the customer's mobile phone but not operating it, thereby improving the accuracy of rule-breaking operation detection.

[0121] It should be noted that in this embodiment, each model installed in the edge computing device, such as the target recognition model and the staff action analysis model, can use an (Open Neural Network Exchange, ONNX) format model, which can better achieve software and hardware separation. The model and algorithm are adapted to different edge devices and deployed in the edge devices. The camera video data stream is obtained, the recognition range is manually divided according to different scenes, and the recognized rule-breaking information is pushed to the management platform.

[0122] In the embodiment, video stream data collected by multiple camera devices is acquired, target recognition operation is performed on the video stream data, and personnel detection results and article detection results are obtained; the personnel detection results include personnel categories and human body key point positions of each personnel; the article detection results include types and positions of articles; if a personnel hand position in a human body key point position of a first type of personnel in the personnel detection results is empty, a personnel hand position of the first type of personnel in the video stream data is predicted using personnel detection results corresponding to historical video stream data, and three-dimensional coordinates of the personnel hand position of the first type of personnel are determined according to personnel hand positions of the first type of personnel corresponding to each of the video stream data. In addition, a trajectory of a target device of a second type of personnel is analyzed based on article detection results corresponding to each of the video stream data, two-dimensional coordinates of the target device in the video stream data are obtained, and three-dimensional coordinates of the target device are determined according to the two-dimensional coordinates of the target device in the video stream data. That is, through the above steps, the three-dimensional coordinates of the hands of the staff and the three-dimensional coordinates of the customer's mobile phone are detected, the illegal customer operation warning information can be determined when the relative distance between the three-dimensional coordinates of the hands of the staff and the three-dimensional coordinates of the customer's mobile phone is not in compliance, the detection of the customer operation mobile phone is realized, and the entire process is analyzed in real time through the video stream, without the need for manual participation, and manpower is saved.

[0123] In addition, in the embodiment, when a personnel hand position of a first type of personnel is not detected in the personnel detection results, a personnel hand position of the first type of personnel in the video stream data is predicted using personnel detection results corresponding to historical video stream data, the hand position of the staff can be predicted when the hand of the staff is blocked, it is ensured that the hand position of the staff can be obtained in real time, and the reliability and accuracy of the detection of the customer operation mobile phone are improved.

[0124] In addition, in the embodiment, based on article detection results corresponding to each of the video stream data, a trajectory of a target device of a second type of personnel is analyzed, two-dimensional coordinates of the target device in the video stream data are obtained, and three-dimensional coordinates of the target device are determined according to the two-dimensional coordinates of the target device in the video stream data, the motion trajectory of the user's mobile phone can be monitored in real time, the target can be prevented from being lost in the process of handover, and the accuracy and reliability of the detection of the customer operation mobile phone are improved.

[0125] Another embodiment of the present application provides a specific implementation of “predicting a personnel hand position of a first type of personnel in the video stream data using personnel detection results corresponding to historical video stream data”, which can include the following steps:

[0126] 1) From the human key point position in the personnel detection result corresponding to the historical video stream, the position of each key point of the first type of personnel is screened out.

[0127] Specifically, for the video stream data in which the hand position of the staff is not recognized, when the hand is occluded for a period of time, the occluded hand position can be predicted by using time information and motion trajectory, which is a method of predicting the overlapping joint node in the future frame by analyzing the continuity of motion and historical information. The core idea of this method is that human motion usually has continuity in the time dimension, especially the motion of arms and hands does not suddenly deviate. Even if the hand is occluded in the first frame, the motion direction and position of the hand can be reasonably inferred by combining the key points such as shoulder, elbow, wrist motion trajectory, and the corresponding known information of the previous several frames.

[0128] Based on the personnel detection result corresponding to the historical video stream data collected by the camera, the positions of the 16 key points of the staff in each frame are accurately positioned in the frames in which the hand is not occluded, and the record information of the key points in the time dimension is established.

[0129] 2) Based on the logical relationship between the non-hand key point position and the hand key point position, the position of each key point of the first type of personnel is used to predict the hand position of the first type of personnel in the video stream data.

[0130] In this embodiment, the logical relationship between the non-hand key point position and the hand key point position can be determined by using a time series model, that is, a model for predicting the hand key point position based on the non-hand key point position is pre-trained, the model is internally configured with the logical relationship between the non-hand key point position and the hand key point position, and then the position of the hand of the staff in the current video stream data is predicted by using the position of each key point of the first type of personnel in the historical video stream.

[0131] Among them, time series modeling is the basis for analyzing the change of hand motion in continuous frames. The key point position information of the past several frames can be combined as a time series to predict the position of the next frame by using a long short-term memory network (LSTM, Long Short-Term Memory). First, the required LSTM network needs to be trained, which is as follows:

[0132] The data prepared during training includes:

[0133] The time series data of the human key points, especially the change of the position of the key points with time (coordinate sequence of shoulders, elbows, wrists, and fingers).

[0134] Input data format: The key point data of each frame is expressed in two-dimensional coordinates, so the data of a single frame can be represented as a set of two-dimensional coordinates of each key point.

[0135] The structure of the LSTM model used for training is as follows:

[0136] The structure of the LSTM is composed of multiple "memory cells" that can dynamically remember or forget information in the time series based on the input data. Information flow is controlled through input, forget, and output gates.

[0137] 1. Input layer: The input time series keypoint data is passed into the LSTM network.

[0138] Each input data point is a multi-dimensional vector (e.g., joint coordinates of a frame) that is continuously inputted along the time axis.

[0139] 2. LSTM layer:

[0140] Input gate: Determines the importance of the current input and whether to pass the keypoint positions of the current frame to the memory cells.

[0141] Forget gate: Determines whether to forget some information from the past, for example, if the hand motion changes, the LSTM can choose to "forget" the hand position from a long time ago.

[0142] Output gate: Based on the current input and past memory, it decides what to output, i.e., predicts the position of the hand in the next frame.

[0143] The LSTM network extracts the rules of hand motion from the input time series through these gating mechanisms. It remembers the motion patterns of the past few frames and, through this time-dependent relationship, predicts the motion trend of future frames.

[0144] 3. Fully connected layer: The output of the LSTM is passed to the fully connected layer, which is used to convert the hidden state of the LSTM into the predicted keypoint positions. The output of this layer is the keypoint coordinates of the next frame.

[0145] In the LSTM model, the mean-square error (MSE) is used as the loss function to measure the difference between the predicted keypoint positions and the actual keypoint positions. In addition, the Adam optimizer is used to minimize the loss function to optimize the model.

[0146] After preparing the data and constructing the model structure, the LSTM model is trained. Specifically, the time series mentioned above, i.e., the positions of 16 key points in a series of consecutive frames, are used as the input of the LSTM to train the model.

[0147] The subsequent LSTM model can combine the input data to predict the hand position in the future frame. For example, the 16 key point positions of the previous frames in the video stream data are input to obtain the hand position in the current frame, so as to predict the hand position information in the occluded frame. In addition, the non-hand key point position of the first type of personnel detected in the current video stream can also be input during prediction, so as to realize accurate prediction.

[0148] In the embodiment, the hand position can be predicted when the hand position of the staff is occluded, sufficient data can be provided for subsequent illegal operation, and the accuracy of illegal operation detection can be improved.

[0149] After obtaining the hand position of the first type of personnel corresponding to each video stream data, the three-dimensional coordinates of the hand position of the first type of personnel can be determined according to the hand position of the first type of personnel corresponding to each video stream data, specifically including:

[0150] 1) Calculate the parallax data of the hand position of the first type of personnel in different video stream data.

[0151] Specifically, due to the limitations of single camera range, angle and other factors, it is difficult to determine the distance between the staff's fingers and the customer's mobile phone. The embodiment of the application designs a three-dimensional distance detection algorithm of multiple cameras, which deduces the three-dimensional coordinates of the object by the method of multiple camera linkage combined with the relative position of the camera and the image shot, and realizes the calculation of the distance between the fingers and the mobile phone.

[0152] When performing specific distance calculation, the camera position needs to be calibrated in advance. First, the position of each camera (camera external parameter) and the internal parameter of the camera (focal length, principal point, etc.) need to be known. These information can be obtained by camera calibration.

[0153] In addition, the position of the object to be detected also needs to be known, which specifically refers to the position of the detected object in the angle of view of each camera, that is, the pixel coordinates of the object, which specifically refers to two-dimensional coordinates. Taking the hand as an example, the position is the hand position of the staff detected above.

[0154] After the calibrated camera data and the hand position of the staff, the three-dimensional distance detection algorithm can be used to determine the three-dimensional coordinates of the hand.

[0155] First, the parallax data of the hand position of the first type of personnel in different video stream data is calculated. When calculating the parallax data, two video stream data collected by the camera devices are needed. In the embodiment of the present application, since multiple camera devices are configured, two video stream data can be selected randomly or according to certain rules from all the video stream data collected by the camera devices, and the parallax is calculated by using the two video stream data, and then the three-dimensional coordinates of the hand are obtained by using the parallax. Alternatively, the parallax is calculated for every two video stream data, and then the corresponding three-dimensional coordinates of the hand are obtained by using the parallax, and the average value of all the three-dimensional coordinates of the hand is calculated.

[0156] When calculating the parallax in detail, for the camera devices with overlapping fields of view, the projection positions of the same object are different when the object is observed from different camera angles, and the parallax refers to the difference of the projection positions.

[0157] 2) According to the parallax data, the relative distance between the hand of the first type of personnel and the corresponding camera device is calculated.

[0158] Specifically, the depth information is derived by using triangulation.

[0159] In detail, according to the parallax formula, the depth information (i.e. the distance of the object to the camera device) of the object, such as the hand of the staff, to the camera device can be calculated. The specific calculation formula is as follows:

[0160] Z = f * B / d

[0161] Wherein, Z is the distance (depth information) of the object to the camera device, f is the focal length of the camera device, B is the distance between the two camera devices, and d is the parallax, i.e. the pixel difference of the projection position of the object in the images of the two camera devices.

[0162] The distance of the hand of the staff to the camera device can be calculated by substituting the parallax into the formula Z = f * B / d.

[0163] 3) The three-dimensional coordinates of the hand position of the first type of personnel are calculated by using the relative distance.

[0164] Specifically, by using the relative distance, the three-dimensional distance detection algorithm can be used to obtain the three-dimensional coordinates of the hand position of the first type of personnel, i.e. the position of the hand of the staff in the actual three-dimensional space.

[0165] In the embodiment, the three-dimensional space coordinates of each object can be derived according to the pixel coordinates of the object under each camera device in combination with the triangulation method, so as to obtain the position of the hand of the staff in the actual three-dimensional space, thereby performing subsequent violation judgment.

[0166] Another embodiment of the present application provides a specific implementation of "analyzing the trajectory of the target device of the second type of personnel based on the detection result of the corresponding object in each video stream data, to obtain the two-dimensional coordinates of the target device in the video stream data", referring to Figure 5 may include:

[0167] S21, taking the target device of the second type of personnel as a to-be-tracked target.

[0168] Specifically, due to the limited monitoring range of each monitoring device, object occlusion, insufficient camera line-of-sight range, etc., as the personnel move or hand over the mobile phone, there may be a customer mobile phone in the video stream of one camera device, but at the next moment, the mobile phone disappears in the video stream and appears in the video stream of another camera device. Therefore, in order to reduce target loss, the present application designs a multi-camera linkage multi-target tracking algorithm, which focuses on the association between targets in multiple camera devices, especially when a target disappears after appearing in one camera and appears in another camera, these targets are regarded as the same individual, and the continuous tracking of the customer mobile phone is realized to determine which mobile phone appearing in the video stream is the customer mobile phone that needs to be monitored.

[0169] In a specific implementation, the target device (such as a mobile phone) of the second type of personnel, i.e., a customer, is taken as a to-be-tracked target.

[0170] In an actual scene, in the determination of the to-be-tracked target, the second type of personnel can be filtered out from the personnel detection result corresponding to the video stream data, i.e., the required monitored customer is filtered out, and further, the device closest to the second type of personnel when the device first appears is taken as the target device, and the target device is taken as the to-be-tracked target.

[0171] Specifically, generally, the mobile phone may be held by the customer or placed in a bag. When the appearance of the mobile phone is identified, the center point of the mobile phone and the center point of each personnel are calculated in real time based on the video stream collected by the camera device. The personnel closest to the mobile phone is identified as the owner of the mobile phone, and this information is maintained in a mapping table. Multiple camera devices jointly maintain a mapping table, which stores the correspondence between personnel and mobile phones, such as A mobile phone belongs to XX personnel and B mobile phone belongs to YY personnel.

[0172] Through the above method, the mobile phone of the customer in the personnel detection result can be determined, and the mobile phone is taken as a to-be-tracked target for subsequent target continuous tracking operation.

[0173] S22, determining the appearance feature of the object in the video stream data according to the corresponding object detection result.

[0174] Specifically, when tracking the target, the similarity of the objects can be measured to determine which object is the customer's mobile phone. When measuring the similarity, appearance features can be extracted, and the object detection result can be detected, which includes the object type and the location, the object is labeled by a bounding box at the location of the object, and there is a corresponding type label (also known as a class label).

[0175] Based on the object detection result, the appearance features of the object in the bounding box are extracted. Taking the mobile phone as an example, the appearance features can include the shape of the mobile phone shell, the color, the size of the mobile phone, etc.

[0176] When extracting the appearance features, a feature extractor can be used for feature extraction. The feature extractor is a deep convolutional neural network (CNN), which is responsible for mapping each detected target image segment (bounding box) to a feature vector (usually a high-dimensional feature embedding), which is the extracted appearance feature.

[0177] S23, extract the appearance features of the target to be tracked.

[0178] Similarly, for the target to be tracked, a feature extractor can also be used for feature extraction.

[0179] S24, based on the similarity of the appearance features of the object and the appearance features of the target to be tracked, determine the trajectory of the target to be tracked to obtain the two-dimensional coordinates of the target device in the video stream data.

[0180] Specifically, when tracking, a feature matching and association method is used. In different frames or across cameras, feature matching and association are performed based on appearance features. The essence of feature matching and association is to match the same target at different times or in different cameras by measuring the similarity.

[0181] The similarity of the appearance features of the object and the appearance features of the target to be tracked in different frames or different cameras is calculated. If the similarity is higher than a predetermined threshold, the two targets are considered to be the same target, i.e., the current analyzed object is the customer's mobile phone. If it is lower than the threshold, it is considered to be a different target, i.e., the current analyzed object is not the customer's mobile phone.

[0182] After feature matching, the detection results of the same target in consecutive frames or different camera perspectives can be associated to generate the trajectory of the target. Specifically, in consecutive frames, the target is associated by comparing the appearance features and spatial positions of the target to generate the trajectory. In a multi-camera system, the trajectory of the target can be connected across multiple cameras.

[0183] When there are multiple customer mobile phones in the scene, multiple customer mobile phones are tracked at the same time, and the trajectories of different individuals are ensured not to be confused by distinguishing the appearance characteristics of each target.

[0184] The tracking result for each customer mobile phone, such as the position in the video stream, can be recorded in the mapping table described above, such as the customer mobile phone appearing at position a (specifically a two-dimensional coordinate) in the video collected by camera A and being adjusted to position b in the video collected by camera B. As the position of the customer mobile phone is continuously adjusted, the position of the customer mobile phone in the mapping table is also continuously updated.

[0185] After determining the two-dimensional coordinates of the mobile phone in the video stream data, the two-dimensional coordinates are converted to obtain three-dimensional coordinates, and then based on the relative distance between the three-dimensional coordinates of the hands of the personnel of the first type and the three-dimensional coordinates of the target device, the warning information of the illegal customer operation is determined.

[0186] In the embodiment, after the customer mobile phone is identified, the mobile phone can be tracked in real time to avoid the problem of mobile phone handover, such as user A giving the mobile phone to user B, resulting in the loss of identification of the mobile phone of user A, and the problem of target loss across cameras can also be avoided, that is, the multi-target tracking algorithm of multi-camera linkage solves the problem of target loss.

[0187] In addition, a mapping table is maintained by multiple cameras, the relationship between personnel and mobile phones is maintained by using the mapping table, the matching problem of mobile phones and personnel is solved, and the correspondence relationship between mobile phones and personnel and the position of the mobile phone are recorded in real time.

[0188] In summary, through the multi-target tracking algorithm of multi-camera linkage, the mapping table of the relationship between personnel and mobile phones maintained by multiple cameras, and the combination of multi-camera linkage, the distance calculation between the fingers of the staff and the customer mobile phone is realized, and when the distance between the fingers of the staff and the customer mobile phone screen is less than a certain threshold value for a certain time and there is obvious mobile phone operation, it is judged that the staff operates the mobile phone of the customer in violation of the rules.

[0189] The application automatically checks the illegal operation by using a visual model and an edge computing algorithm, can check the operation of the mobile phone in real time, and timely discovers abnormal behavior and illegal operation. The application greatly reduces the possibility of manual intervention and misjudgment, and can also statistically analyze the detection data to provide decision support and supervision basis for bank management.

[0190] The application tracks the customer mobile phone, can accurately identify the user mobile phone even if there is mobile phone handover, and in the scene where the staff and the customer both have mobile phones, the correspondence between the mobile phone and the personnel can also be accurately realized.

[0191] The above introduces a method for detecting illegal customer operation provided by the embodiment of the application. The following introduces a device for executing the method for detecting illegal customer operation.

[0192] Please refer to Figure 6 , Figure 6 The structure diagram of a device for detecting illegal customer operation provided by the embodiment of the application.

[0193] As Figure 6 shown, the device for detecting illegal customer operation comprises:

[0194] A data acquisition module 11 is configured to acquire video stream data collected by a plurality of camera devices;

[0195] A target identification module 12 is configured to perform target identification on the video stream data to obtain personnel detection results and article detection results. The personnel detection results comprise personnel categories and body key point positions of each personnel. The article detection results comprise types and positions of articles.

[0196] A position prediction module 13 is configured to, if a personnel hand position in a body key point position of a first type of personnel in the personnel detection results is empty, predict the personnel hand position of the first type of personnel in the video stream data by using personnel detection results corresponding to historical video stream data.

[0197] A first coordinate determination module 14 is configured to determine three-dimensional coordinates of the personnel hand position of the first type of personnel according to the personnel hand position of the first type of personnel corresponding to each of the video stream data.

[0198] A second coordinate determination module 15 is configured to analyze a trajectory of a target device of a second type of personnel based on the article detection results corresponding to each of the video stream data to obtain two-dimensional coordinates of the target device in the video stream data.

[0199] A third coordinate determination module 16 is configured to determine three-dimensional coordinates of the target device according to the two-dimensional coordinates of the target device in the video stream data.

[0200] An illegal operation detection module 17 is configured to determine illegal customer operation warning information based on a relative distance between the three-dimensional coordinates of the personnel hand position of the first type of personnel and the three-dimensional coordinates of the target device.

[0201] In a possible implementation, the position prediction module 13 comprises:

[0202] A screening sub-module is configured to screen positions of each key point of the first type of personnel from the body key point positions in the personnel detection results corresponding to the historical video stream data.

[0203] a prediction submodule configured to predict a hand position of the first type of person in the video stream data based on a logical relationship between non-hand key point positions and hand key point positions and positions of respective key points of the first type of person.

[0204] In a possible implementation, the first coordinate determination module 14 includes:

[0205] a parallax calculation submodule configured to calculate parallax data of the hand position of the first type of person in different video stream data;

[0206] a distance calculation submodule configured to calculate a relative distance between the hand of the first type of person and a corresponding camera according to the parallax data;

[0207] a coordinate calculation submodule configured to calculate a three-dimensional coordinate of the hand position of the first type of person by using the relative distance.

[0208] In a possible implementation, the second coordinate determination module 15 includes:

[0209] a target determination submodule configured to determine a target device of a second type of person as a to-be-tracked target;

[0210] a feature determination submodule configured to determine an appearance feature of an article in the video stream data according to a corresponding article detection result;

[0211] a feature extraction submodule configured to extract the appearance feature of the to-be-tracked target;

[0212] a trajectory processing submodule configured to determine a trajectory of the to-be-tracked target based on a similarity between the appearance feature of the article and the appearance feature of the to-be-tracked target, to obtain a two-dimensional coordinate of the target device in the video stream data.

[0213] In a possible implementation, the target determination submodule is specifically configured to:

[0214] screen the second type of person from a person detection result corresponding to the video stream data, determine a device closest to the second type of person when the device first appears as the target device, and determine the target device as the to-be-tracked target.

[0215] In a possible implementation, the violation detection module 17 includes:

[0216] a distance calculation submodule configured to calculate a Euclidean distance between the three-dimensional coordinate of the hand position of the first type of person and the three-dimensional coordinate of the target device;

[0217] The violation detection submodule is configured to determine the violation of the customer operation warning information as an identifier representing a violation if the relative distance is less than the preset threshold for a time period greater than a preset time period and the first type of personnel operates the target device.

[0218] In this embodiment, video stream data collected by multiple camera devices is acquired, target recognition is performed on the video stream data to obtain personnel detection results and article detection results; the personnel detection results include the personnel categories and the human body key point positions of each personnel; the article detection results include the types and positions of articles; if the personnel hand position in the human body key point position of the first type of personnel in the personnel detection results is empty, the personnel hand position of the first type of personnel in the video stream data is predicted by using the personnel detection results corresponding to historical video stream data, and the three-dimensional coordinates of the personnel hand position of the first type of personnel are determined according to the personnel hand positions of the first type of personnel corresponding to each video stream data. In addition, the trajectory of the target device of the second type of personnel is analyzed based on the article detection results corresponding to each video stream data to obtain the two-dimensional coordinates of the target device in the video stream data, and the three-dimensional coordinates of the target device are determined according to the two-dimensional coordinates of the target device in the video stream data. That is, by the above steps, the three-dimensional coordinates of the hands of the staff and the three-dimensional coordinates of the customer's mobile phone are detected, the violation of the customer operation warning information is determined when the relative distance between the three-dimensional coordinates of the hands of the staff and the three-dimensional coordinates of the customer's mobile phone is not in compliance, the detection of the customer operation of the mobile phone is realized, and the whole process is analyzed in real time by the video stream without the need for manual participation, saving manpower. In addition, in this embodiment, when the personnel hand position of the first type of personnel is not detected in the personnel detection results, the personnel hand position of the first type of personnel in the video stream data is predicted by using the personnel detection results corresponding to historical video stream data, the hand position of the staff can be predicted when the hands of the staff are blocked, ensuring that the hand position of the staff can be obtained in real time, and the reliability and accuracy of the detection of the customer operation of the mobile phone are improved. In addition, in this embodiment, the trajectory of the target device of the second type of personnel is analyzed based on the article detection results corresponding to each video stream data to obtain the two-dimensional coordinates of the target device in the video stream data, and the three-dimensional coordinates of the target device are determined according to the two-dimensional coordinates of the target device in the video stream data, which can monitor the motion trajectory of the user's mobile phone in real time, avoid the target from being lost during the handover process, and improve the accuracy and reliability of the detection of the customer operation of the mobile phone.

[0219] It should be noted that the working processes of each module and submodule in this embodiment are described above with reference to the corresponding processes in the above embodiments, and will not be described in detail.

[0220] The electronic device provided in the embodiments of the present application comprises at least one processor and a memory connected with the processor, wherein:

[0221] The memory is configured to store a computer program.

[0222] The processor is configured to execute the computer program, so that the electronic device can implement the above-mentioned method for detecting illegal customer operation.

[0223] Reference Figure 7 As shown in the figure, a structure diagram of an electronic device suitable for implementing the electronic device in the embodiments of the present application is shown. The electronic device in the embodiments of the present application can include, but is not limited to, a fixed terminal such as a mobile phone, a notebook computer, a PDA (Personal Digital Assistant), a PAD (Tablet Personal Computer), a desktop computer, etc. Figure 7 The electronic device shown is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present application.

[0224] As Figure 7 shown, the electronic device can include a processing device (such as a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 602 or programs loaded from a storage device 608 into a random access memory (RAM) 603. In the state that the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected with each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0225] Generally, the following devices can be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device with various devices is shown, but it should be understood that it is not required to implement or have all the shown devices. More or fewer devices can be alternatively implemented or provided.

[0226] The embodiments of the present application also provide a computer program product comprising computer readable instructions, which, when executed on an electronic device, cause the electronic device to implement any one of the methods for detecting illegal customer operation provided in the embodiments of the present application.

[0227] The embodiment of the present application further provides a computer readable storage medium, the storage medium carries one or more computer programs, when the one or more computer programs are executed by an electronic device, the electronic device can realize any kind of illegal customer operation detection method provided by the embodiment of the present application.

[0228] In addition, it should be noted that the above-described device embodiments are only schematic, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e., they may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment. In addition, the connection relationship between the modules in the device embodiment provided by the present application indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.

[0229] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be realized by means of software and necessary general hardware, and of course can also be realized by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. Generally, functions completed by computer programs can be easily realized by corresponding hardware, and the specific hardware structure for realizing the same function can also be various, such as analog circuit, digital circuit or special circuit, etc. However, for the present application, software program implementation is a better embodiment. Based on this understanding, the technical solutions of the present application can be embodied in the form of software products, which are stored in readable storage media, such as computer floppy disks, U disks, mobile hard disks, ROM, RAM, magnetic or optical disks, etc., including a plurality of instructions for making a computer device (which can be a personal computer, a training device, or a network device, etc.) execute the methods described in various embodiments of the present application.

[0230] In the above embodiments, all or part can be realized by software, hardware, firmware or any combination thereof. When realized by software, it can be realized in the form of computer program product in whole or in part.

[0231] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, training device or data center to another website, computer, training device or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be stored by the computer or a data storage device such as a training device, a data center, etc. integrated with one or more available media sets. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

Claims

1. A method of detecting a violation of a customer operation, characterized by, The method comprises: acquiring video stream data collected by multiple cameras; performing target recognition on the video stream data to obtain personnel detection results and article detection results; the personnel detection results comprise personnel categories and body key point positions of each personnel; and the article detection results comprise types and positions of articles; if a personnel hand position in a body key point position of a first type of personnel in the personnel detection results is empty, predicting a personnel hand position of the first type of personnel in the video stream data by using personnel detection results corresponding to historical video stream data; determining three-dimensional coordinates of the personnel hand position of the first type of personnel according to the personnel hand position of the first type of personnel corresponding to each of the video stream data; analyzing a trajectory of a target device of a second type of personnel based on the article detection results corresponding to each of the video stream data to obtain two-dimensional coordinates of the target device in the video stream data; determining three-dimensional coordinates of the target device according to the two-dimensional coordinates of the target device in the video stream data; determining a warning information of a rule violation of a customer operation based on a relative distance between the three-dimensional coordinates of the personnel hand position of the first type of personnel and the three-dimensional coordinates of the target device.

2. The method of detecting rogue customer operations of claim 1, wherein, The method of predicting the personnel hand position of the first type of personnel in the video stream data by using the personnel detection results corresponding to the historical video stream data comprises: screening positions of each key point of the first type of personnel from the body key point positions in the personnel detection results corresponding to the historical video stream data; predicting the personnel hand position of the first type of personnel in the video stream data by using the positions of each key point of the first type of personnel based on a logical relationship between non-hand key point positions and hand key point positions.

3. The method of detecting a rogue operation according to claim 1, wherein The method of determining the three-dimensional coordinates of the personnel hand position of the first type of personnel according to the personnel hand position of the first type of personnel corresponding to each of the video stream data comprises: calculating parallax data of the personnel hand position of the first type of personnel in different video stream data; calculating a relative distance between the personnel hand of the first type of personnel and a corresponding camera according to the parallax data; calculating the three-dimensional coordinates of the personnel hand position of the first type of personnel by using the relative distance.

4. The method of detecting a rogue operation according to claim 1, wherein The method of analyzing the trajectory of the target device of the second type of personnel based on the article detection results corresponding to each of the video stream data to obtain the two-dimensional coordinates of the target device in the video stream data comprises: regarding the target device of the second type of personnel as a to-be-tracked target; determining appearance features of articles in the video stream data according to the corresponding article detection results; extracting appearance features of the to-be-tracked target; determining the trajectory of the to-be-tracked target based on a similarity between the appearance features of the articles and the appearance features of the to-be-tracked target to obtain the two-dimensional coordinates of the target device in the video stream data.

5. The method of detecting a rogue operation according to claim 4, wherein, The method of regarding the target device of the second type of personnel as the to-be-tracked target comprises: screening the second type of personnel from the personnel detection results corresponding to the video stream data; regarding a device closest to the second type of personnel when the device first appears as the target device. The target device is taken as a to-be-tracked target.

6. The method of detecting a rogue operation of claim 1, wherein, The relative distance between the three-dimensional coordinates of the hand position of the first type of personnel and the three-dimensional coordinates of the target device is determined, and a warning information of illegal customer operation is determined based on the relative distance. The Euclidean distance between the three-dimensional coordinates of the hand position of the first type of personnel and the three-dimensional coordinates of the target device is calculated. If the relative distance is less than a preset threshold for a time greater than a preset time and there is a behavior of the first type of personnel operating the target device, the warning information of illegal customer operation is determined as an identification representing illegal operation.

7. A device for detecting a violation of a customer operation, characterized by comprising: The method comprises the following steps: The data acquisition module is configured to acquire video stream data collected by a plurality of camera devices. The target identification module is configured to perform target identification on the video stream data to obtain personnel detection results and article detection results. The position prediction module is configured to, if the personnel detection results do not include the hand position of the first type of personnel, predict the hand position of the first type of personnel in the video stream data based on historical personnel detection results corresponding to the video stream data. The first coordinate determination module is configured to determine the three-dimensional coordinates of the hand position of the first type of personnel based on the hand position of the first type of personnel corresponding to each of the video stream data. The second coordinate determination module is configured to analyze the trajectory of a target device of a second type of personnel based on the article detection results corresponding to each of the video stream data to obtain two-dimensional coordinates of the target device in the video stream data. The third coordinate determination module is configured to determine the three-dimensional coordinates of the target device based on the two-dimensional coordinates of the target device in the video stream data. The illegal operation detection module is configured to determine a warning information of illegal customer operation based on the relative distance between the three-dimensional coordinates of the hand position of the first type of personnel and the three-dimensional coordinates of the target device.

8. A computer program product, characterised in that, The computer readable instructions, when executed on an electronic device, cause the electronic device to implement the method for detecting illegal customer operation according to any one of claims 1 to 6.

9. An electronic device, comprising: The memory is configured to store a computer program. The processor is configured to execute the computer program to enable the electronic device to implement the method for detecting illegal customer operation according to any one of claims 1 to 6. The storage medium carries one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the method for detecting illegal customer operation according to any one of claims 1 to 6.

10. A computer storage medium, characterized in that, ​

Images