Detection method for illegal valet operation and related device

Abstract

The invention discloses an illegal valet operation detection method and a related device, which can be applied to the field of artificial intelligence or finance. Obtaining video stream data collected by the camera device, carrying out target identification operation on the video stream data to obtain a personnel detection result and an article detection result, and if the personnel detection result shows that the personnel hand position of the first type of personnel is empty, carrying out prediction operation on the personnel hand position of the first type of personnel to obtain an article detection result; and according to the personnel hand positions of the first type of personnel corresponding to the video stream data, determining three-dimensional coordinates of the personnel hand positions of the first type of personnel. In addition, three-dimensional coordinates of the target device are determined. According to the method, the three-dimensional coordinates of the hands of the worker and the three-dimensional coordinates of the mobile phone of the client are detected through the steps, so that the illegal valet operation early warning information can be determined when the relative distance between the three-dimensional coordinates of the hands of the worker and the three-dimensional coordinates of the mobile phone of the client is not compliant, and the detection of valet operation on the mobile phone is realized.

Description

Technical Field

[0001] The present application relates to the field of financial technology, and in particular to a method for detecting illegal proxy operations and related devices. Background Art

[0002] Operating a mobile phone on behalf of a customer refers to the phenomenon where a bank staff uses a customer's mobile phone to complete certain operations when the customer needs help or for convenience. This behavior is considered a violation of regulations at bank branches.

[0003] Operating a mobile phone on behalf of a customer can bring many risks, such as customer information leakage, staff stealing customer funds, and violation of laws and regulations. To avoid such violations, the conventional inspection methods are manual inspections and surveillance video spot checks, but these methods have low detection accuracy. Summary of the invention

[0004] In view of the above problems, the present application provides a detection method and related device for illegal valet operation, so as to achieve the purpose of improving the detection accuracy of valet operation of mobile phones. The specific scheme is as follows:

[0005] The first aspect of the present application provides a method for detecting illegal valet operation, comprising:

[0006] Acquire video stream data collected by multiple camera devices;

[0007] Performing target recognition operation on the video stream data to obtain personnel detection results and object detection results; the personnel detection results include the personnel category and the position of key points of each person; the object detection results include the type and position of the object;

[0008] If the position of the person's hand in the key point position of the first type of person in the person detection result is empty, predict the position of the person's hand of the first type of person in the video stream data by using the person detection result corresponding to the historical video stream;

[0009] Determine the three-dimensional coordinates of the hand positions of the first type of personnel according to the hand positions of the first type of personnel corresponding to each of the video stream data;

[0010] Based on the object detection results corresponding to each of the video stream data, analyzing the trajectory of the target device of the second type of person to obtain the two-dimensional coordinates of the target device in the video stream data;

[0011] Determining the three-dimensional coordinates of the target device according to the two-dimensional coordinates of the target device in the video stream data;

[0012] Based on the three-dimensional coordinates of the hand position of the first type of personnel and the relative distance of the three-dimensional coordinates of the target device, the warning information of illegal valet operation is determined.

[0013] In a possible implementation, predicting the hand position of the first type of person in the video stream data by using the person detection result corresponding to the historical video stream includes:

[0014] Filtering out the positions of the key points of the first type of people from the positions of the key points of the human body in the personnel detection results corresponding to the historical video stream;

[0015] Based on the logical relationship between the non-hand key point positions and the hand key point positions, the positions of the key points of the first type of personnel are used to predict the hand positions of the first type of personnel in the video stream data.

[0016] In a possible implementation, determining the three-dimensional coordinates of the hand positions of the first type of personnel according to the hand positions of the first type of personnel corresponding to each of the video stream data includes:

[0017] Calculating disparity data of the hand positions of the first type of personnel in different video stream data;

[0018] Calculating the relative distance between the hand of the first type of person and the corresponding camera device according to the parallax data;

[0019] The three-dimensional coordinates of the hand position of the first type of person are calculated using the relative distance.

[0020] In a possible implementation, based on the object detection results corresponding to each of the video stream data, analyzing the trajectory of the target device of the second type of person to obtain the two-dimensional coordinates of the target device in the video stream data includes:

[0021] taking the target device of the second type of person as a target to be tracked;

[0022] Determining appearance features of objects in the video stream data according to corresponding object detection results;

[0023] Extracting appearance features of the target to be tracked;

[0024] Based on the similarity between the appearance features of the object and the appearance features of the target to be tracked, the trajectory of the target to be tracked is determined to obtain the two-dimensional coordinates of the target device in the video stream data.

[0025] In a possible implementation, taking the target device of the second type of person as the target to be tracked includes:

[0026] Screening out a second type of person from the person detection results corresponding to the video stream data;

[0027] The device that is closest to the second type of person when the device first appears is used as the target device;

[0028] The target device is used as a target to be tracked.

[0029] In a possible implementation, determining the warning information of illegal valet operation based on the three-dimensional coordinates of the hand position of the first type of person and the relative distance of the three-dimensional coordinates of the target device includes:

[0030] Calculating the Euclidean distance between the three-dimensional coordinates of the hand position of the first type of person and the three-dimensional coordinates of the target device;

[0031] If the time during which the relative distance is less than the preset threshold is greater than the preset time and there is an act of the first type of person operating the target device, it is determined that the warning information of the illegal valet operation is an identification indicating a violation.

[0032] The second aspect of the present application provides a detection device for illegal valet operation, comprising:

[0033] A data acquisition module, used to acquire video stream data collected by multiple camera devices;

[0034] A target recognition module is used to perform a target recognition operation on the video stream data to obtain a person detection result and an object detection result; the person detection result includes the person category and the position of key points of each person; the object detection result includes the type and position of the object;

[0035] A position prediction module, used for predicting the position of the hand of the first type of person in the video stream data by using the person detection result corresponding to the historical video stream if the position of the hand of the first type of person in the key point position of the human body in the person detection result is empty;

[0036] A first coordinate determination module, configured to determine the three-dimensional coordinates of the hand positions of the first type of personnel according to the hand positions of the first type of personnel corresponding to each of the video stream data;

[0037] A second coordinate determination module, configured to analyze the trajectory of the target device of the second type of person based on the object detection results corresponding to each of the video stream data, so as to obtain the two-dimensional coordinates of the target device in the video stream data;

[0038] A third coordinate determination module, used to determine the three-dimensional coordinates of the target device according to the two-dimensional coordinates of the target device in the video stream data;

[0039] The violation detection module is used to determine the warning information of the illegal valet operation based on the three-dimensional coordinates of the hand position of the first type of personnel and the relative distance of the three-dimensional coordinates of the target device.

[0040] According to a third aspect of the present application, a computer program product is provided, comprising computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the above-mentioned method for detecting illegal valet operations.

[0041] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0042] The memory is used to store computer programs;

[0043] The processor is used to execute the computer program so that the electronic device can implement the above-mentioned method for detecting illegal valet operations.

[0044] A fifth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the above-mentioned method for detecting illegal valet operations.

[0045] By means of the above technical scheme, the present application provides a detection method and related device for illegal valet operation. In the present application, video stream data collected by multiple camera devices is obtained, and target recognition operation is performed on the video stream data to obtain personnel detection results and object detection results; the personnel detection results include the personnel category and the key point position of each person; the object detection results include the type and position of the object. If the personnel hand position of the first type of personnel in the key point position of the human body is empty in the personnel detection results, the personnel detection results corresponding to the historical video stream are used to predict the personnel hand position of the first type of personnel in the video stream data, and the three-dimensional coordinates of the personnel hand position of the first type of personnel are determined according to the personnel hand position of the first type of personnel corresponding to each of the video stream data. In addition, based on the object detection results corresponding to each of the video stream data, the trajectory of the target device of the second type of personnel is analyzed to obtain the two-dimensional coordinates of the target device in the video stream data, and the three-dimensional coordinates of the target device are determined according to the two-dimensional coordinates of the target device in the video stream data. That is, the present invention detects the three-dimensional coordinates of the staff's hands and the three-dimensional coordinates of the customer's mobile phone through the above steps, and can determine the warning information of illegal valet operation when the relative distance between the three-dimensional coordinates of the staff's hand position and the three-dimensional coordinates of the customer's mobile phone is not in compliance, thereby realizing the detection of valet operation of mobile phones, and the entire process is analyzed in real time through video streams, without manual participation, saving manpower. In addition, in the present invention, when the hand position of the first type of personnel is not detected in the personnel detection results, the personnel detection results corresponding to the historical video stream are used to predict the hand position of the first type of personnel in the video stream data, and the hand position of the staff can be predicted when the staff's hand is blocked, ensuring that the staff's hand position can be obtained in real time, thereby improving the reliability and accuracy of valet operation of mobile phones. In addition, in the present invention, based on the object detection results corresponding to each of the video stream data, the trajectory of the target device of the second type of personnel is analyzed to obtain the two-dimensional coordinates of the target device in the video stream data, and the three-dimensional coordinates of the target device in the video stream data are determined. This can monitor the movement trajectory of the user's mobile phone in real time to avoid losing the target during the handover process of the user's mobile phone, and can also improve the accuracy and reliability of the detection of the mobile phone operated on behalf of the customer. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0047] Figure 1A schematic diagram of a system architecture provided for this application;

[0048] Figure 2 A schematic diagram of a terminal structure provided for this application;

[0049] Figure 3 A schematic diagram of the structure of an edge computing device provided for this application;

[0050] Figure 4 A flow chart of a method for detecting illegal valet operations provided by this application;

[0051] Figure 5 A flowchart of a method for determining coordinates provided in this application;

[0052] Figure 6 A schematic diagram of the structure of a detection device for illegal valet operation provided in this application;

[0053] Figure 7 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0054] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0055] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0056] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0057] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0058] Operating a mobile phone on behalf of a customer refers to the phenomenon where a bank staff uses a customer's mobile phone to complete certain operations when the customer needs help or for convenience. This behavior is considered a violation of regulations at bank branches.

[0059] Operating a mobile phone on behalf of a customer can bring many risks, such as customer information leakage, staff stealing customer funds, and violation of laws and regulations. To avoid such violations, the conventional inspection method is manual inspection and surveillance video spot check, but these methods are easily subject to human subjective assumptions and have low detection accuracy. In addition, there are also problems of high labor costs and low inspection efficiency.

[0060] In addition, there are also methods that use face matching algorithms to determine the type of person and then determine whether there is any illegal proxy operation. However, in banking scenarios, the camera angle of the mobile phone operation scene needs to be able to see the mobile phone more clearly, so the camera angle is mostly vertical, and it is basically impossible to obtain the person's face information. Therefore, this method of determining the identity of the person is not applicable in the scenario of proxy mobile phone operation.

[0061] In view of the problems of existing methods for checking violations of mobile phone operation on behalf of customers, the embodiments of the present invention use visual models and edge computing algorithms to automatically detect violations of mobile phone operation on behalf of customers, which can check the operation of mobile phones in real time and detect abnormal behaviors and illegal operations in time. The present invention is more in line with actual banking scenarios, greatly reducing the possibility of manual intervention and misjudgment, and can also count and analyze detection data to provide decision support and regulatory basis for bank management.

[0062] Specifically, the visual model and edge computing algorithm are deployed to the edge computing box, and the real-time automatic warning function of illegal behavior is realized through the management platform. The visual model is used to obtain personnel information, including personnel category and personnel action recognition. Combined with these personnel information, the personnel behavior analysis algorithm is used to determine whether there is any staff member operating the mobile phone on behalf of the customer. When a violation occurs, the violation information is pushed to the management platform in a timely manner. The management platform records and counts the violations, notifies relevant personnel in real time to handle the warning information, and realizes the visual automatic inspection of violations.

[0063] In detail, the present application provides a detection method and related device for illegal valet operation. In the present application, video stream data collected by multiple camera devices is obtained, and target recognition operation is performed on the video stream data to obtain personnel detection results and object detection results; the personnel detection results include the personnel category and human body key point positions of each person; the object detection results include the type and position of the object. If the personnel hand position of the first type of personnel in the human body key point position is empty in the personnel detection result, the personnel detection result corresponding to the historical video stream is used to predict the personnel hand position of the first type of personnel in the video stream data, and the three-dimensional coordinates of the personnel hand position of the first type of personnel are determined according to the personnel hand position of the first type of personnel corresponding to each of the video stream data. In addition, based on the object detection results corresponding to each of the video stream data, the trajectory of the target device of the second type of personnel is analyzed to obtain the two-dimensional coordinates of the target device in the video stream data, and the three-dimensional coordinates of the target device are determined according to the two-dimensional coordinates of the target device in the video stream data. That is, the present invention detects the three-dimensional coordinates of the staff's hands and the three-dimensional coordinates of the customer's mobile phone through the above steps, and can determine the warning information of illegal valet operation when the relative distance between the three-dimensional coordinates of the staff's hand position and the three-dimensional coordinates of the customer's mobile phone is not in compliance, thereby realizing the detection of valet operation of mobile phones, and the entire process is analyzed in real time through video streams, without manual participation, saving manpower. In addition, in the present invention, when the hand position of the first type of personnel is not detected in the personnel detection results, the personnel detection results corresponding to the historical video stream are used to predict the hand position of the first type of personnel in the video stream data, and the hand position of the staff can be predicted when the staff's hand is blocked, ensuring that the staff's hand position can be obtained in real time, thereby improving the reliability and accuracy of valet operation of mobile phones. In addition, in the present invention, based on the object detection results corresponding to each of the video stream data, the trajectory of the target device of the second type of personnel is analyzed to obtain the two-dimensional coordinates of the target device in the video stream data, and the three-dimensional coordinates of the target device in the video stream data are determined. This can monitor the movement trajectory of the user's mobile phone in real time to avoid losing the target during the handover process of the user's mobile phone, and can also improve the accuracy and reliability of the detection of the mobile phone operated on behalf of the customer.

[0064] It should be noted that the detection method and related device of illegal valet operation provided by the present invention can be used in the field of artificial intelligence or finance. The above is only an example and does not limit the application field of the detection method and related device of illegal valet operation provided by the present invention.

[0065] See also Figure 1 , Figure 1A schematic diagram of a system architecture is shown. The system may include a terminal 100 and an edge computing device 200. The edge computing device 200 may include one or more edge computing devices ( Figure 1 In the example of an edge computing device, the edge computing device 200 can provide the method provided in the embodiment of the present application for one or more terminals.

[0066] Among them, a video stream acquisition application can be installed on the terminal 100. The above application and web page can provide an interface. The terminal 100 can receive relevant parameters entered by the user on the video stream acquisition interface and send the acquired video stream to the edge computing device 200. The edge computing device 200 can obtain processing results based on the received data.

[0067] Next describe Figure 1 The product form of the mid-terminal 100;

[0068] The terminal 100 in the embodiment of the present application can be a camera device, a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiment of the present application does not impose any limitation on this.

[0069] Figure 2 An optional hardware structure diagram of the terminal 100 is shown.

[0070] refer to Figure 2 As shown, the terminal 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), an earphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190 and other components. Those skilled in the art will appreciate that Figure 2 These are merely examples of terminals or multi-function devices and do not constitute limitations on the terminals or multi-function devices, which may include more or fewer components than those shown in the figures, or combinations of certain components, or different components.

[0071] The input unit 130 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the portable multifunctional device. Specifically, the input unit 130 may include a touch screen 131 (optional) and / or other input devices 132. The touch screen 131 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute the command sent by the processor 170; the touch signal at least includes the touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0072] Among them, the input device 132 can receive input data and the like.

[0073] The display unit 140 may be used to display information input by the user or provided to the user, various menus of the terminal 100, interactive interfaces, file displays, and / or playback of any multimedia file. In an embodiment of the present application, the display unit 140 may be used to display an interface for video stream acquisition, processing results, and the like.

[0074] The memory 120 can be used to store instructions and data. The memory 120 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc.; the instruction storage area can store software units such as operating systems, applications, instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; provide the processor 170 with hardware, software and data resources including management of computing and processing equipment, and support control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0075] The processor 170 is the control center of the terminal 100. It uses various interfaces and lines to connect various parts of the entire terminal 100. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it executes various functions of the terminal 100 and processes data, thereby controlling the terminal device as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 170. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 170 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0076] Among them, the memory 120 can be used to store software codes related to the video stream acquisition method, the processor 170 can execute the steps of the video stream acquisition method, and can also schedule other units (such as the above-mentioned input unit 130 and display unit 140) to achieve corresponding functions.

[0077] The radio frequency unit 110 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 170 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0078] In this embodiment of the present application, the RF unit 110 can send data to the edge computing device 200 and receive processing results sent by the edge computing device 200.

[0079] It should be understood that the radio frequency unit 110 is optional and can be replaced by other communication interfaces, such as a network port.

[0080] The terminal 100 also includes a power supply 190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0081] The terminal 100 further includes an external interface 180 , which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal 100 to communicate with other devices, or to connect a charger to charge the terminal 100 .

[0082] Although not shown, the terminal 100 may also include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which are not described in detail here. Some or all of the methods described below may be applied in the following embodiments. Figure 2 In the terminal 100 shown.

[0083] Next describe Figure 1 The product form of the edge computing device 200;

[0084] Figure 3 A schematic diagram of the structure of an edge computing device 200 is provided, such as Figure 3 As shown, the edge computing device 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate with each other through the bus 201.

[0085] The bus 201 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0086] The processor 202 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0087] The memory 204 may include a volatile memory, such as a random access memory (RAM). The memory 204 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard drive (HDD), or a solid state drive (SSD).

[0088] The memory 204 may be used to store software codes related to the method for detecting illegal valet operations, and the processor 202 may execute the steps of the method for detecting illegal valet operations of the chip, and may also schedule other units to implement corresponding functions.

[0089] It should be understood that the above-mentioned terminal 100 and edge computing device 200 can be centralized or distributed devices, and the processors in the above-mentioned terminal 100 and edge computing device 200 (such as processor 170 and processor 202) can be hardware circuits (such as application specific integrated circuit (ASIC), field-programmable gate array (FPGA), general-purpose processor, digital signal processor (DSP), microprocessor or microcontroller, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0090] In order to solve the above problems, the embodiment of the present application provides a detection method for illegal valet operation. The detection method for illegal valet operation of the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0091] Reference Figure 4 , Figure 4 A flow chart of a method for detecting illegal valet operation provided in an embodiment of the present application is shown as follows: Figure 4 As shown, a method for detecting illegal valet operation provided by an embodiment of the present application may include:

[0092] S11, obtaining video stream data collected by multiple camera devices.

[0093] In actual scenarios, since the bank's working area is large and the monitoring range of each camera device is limited, multiple cameras are needed to monitor the bank's working area. Generally, the camera device can be a camera installed on the roof with a vertical angle.

[0094] Each camera device can collect video stream data in real time and output the collected video stream data to the edge computing device so that the edge computing device can detect illegal valet operations.

[0095] S12, performing a target recognition operation on the video stream data to obtain a person detection result and an object detection result; the person detection result includes the person category and the position of key points of each person; the object detection result includes the type and position of the object.

[0096] Specifically, in actual scenarios, illegal valet operation usually means that the staff operates the customer's device on behalf of the customer, such as mobile phones, tablets, etc. Therefore, it is necessary to identify the staff, customers and objects in the video stream data.

[0097] Taking the mobile phone as an example, the data of operating the mobile phone is obtained, and the public data is integrated as a training set. The target recognition model labels are set to staff, customers and mobile phones, the data is annotated, and the target recognition model is trained.

[0098] After the model training is completed, the model pt file is installed in the edge computing device. When the model is used, the video stream data is input into the model. The model will mark the identified targets with a label box, such as a rectangular box, and obtain a pt file. The pt file includes the identified targets, such as staff, customers, and objects (such as mobile phones).

[0099] When operating on behalf of customers, in addition to the above-mentioned staff, customers and objects (such as mobile phones), there is also a need for the behavior of operating the mobile phone. Therefore, in the embodiment of the present invention, in addition to the need to identify the staff, customers and objects (such as mobile phones), it is also necessary to detect the operating behavior, which is the current action of the person.

[0100] When detecting specific actions, a personnel action analysis model can be used. The personnel action analysis model is different from other models that focus on all 17 key points of the human body. Since the camera angle in the scene of this embodiment is a vertical angle and mainly focuses on hand movements, in this embodiment, the personnel action analysis model mainly focuses on 16 key points of the upper body: shoulder, elbow, wrist, thumb tip thumb_tip, index finger tip index_tip, middle finger tip middle_tip, ring finger tip ring_tip, little finger tip little_tip, and trains 16 human key point models (specifically: shoulder, elbow, wrist and the top of each finger). When the person is blocked or overlapped, it can accurately determine the position of the target person's hand and the movement of the target person's hand, and can effectively analyze the person's hand movement.

[0101] Therefore, the target recognition model and the personnel motion analysis model are called to process the video stream data to obtain the personnel detection results and the object detection results.

[0102] The personnel detection result includes the personnel category and the position of key points of the human body of each person. For example, if the person is a staff member, the position of the staff member's shoulder, elbow, wrist and the top of each finger is marked in the staff member's annotation box, and each key point is marked with a point. The same process is performed for customers.

[0103] The object detection result includes the type and location of the object. If the object is a mobile phone, a marking box is used to mark the location of the mobile phone.

[0104] S13. If the hand position of the first type of personnel in the human body key point position in the personnel detection result is empty, use the personnel detection result corresponding to the historical video stream to predict the hand position of the first type of personnel in the video stream data.

[0105] In practical applications, for each video stream data collected by a camera device, some video stream data may be blocked by objects such as machines and pillars, resulting in failure to detect the hands of the person. In this case, the hand position of the corresponding person can be predicted by a prediction method.

[0106] When illegal valet operations are performed, it is usually the staff who operate the user's mobile phone. Therefore, we only need to pay attention to the hand movements of the staff. For the personnel detection results corresponding to each video stream data, if it is the first type of personnel, specifically the hand position of the staff in the key point position of the human body is empty, that is, the staff's hand is not detected. Then, the position of the hand can be predicted by using the positions of the shoulder, elbow, and wrist in the personnel detection results of the video stream where the hand is detected in the historical video stream collected by the same camera device.

[0107] Among them, if the hand position of a person is detected in some video stream data, there is no need to perform a prediction operation. That is, this embodiment only performs a hand position prediction operation on the video stream data where the hand position of a person is not detected, and finally obtains the hand position of the staff member corresponding to each video stream data.

[0108] S14. Determine the three-dimensional coordinates of the hand positions of the first type of personnel according to the hand positions of the first type of personnel corresponding to each of the video stream data.

[0109] Specifically, the hand position of the first type of personnel is a two-dimensional coordinate in the video stream data, which needs to be converted into three-dimensional space to obtain the three-dimensional coordinates of the person's hand position, that is, to obtain the specific position of the person's hand position in the three-dimensional space.

[0110] Compared with point cloud data, the two-dimensional coordinates and three-dimensional coordinates in this embodiment require less space and can be fully deployed in the edge computing box.

[0111] S15. Analyze the trajectory of the target device of the second type of person based on the object detection results corresponding to each of the video stream data to obtain the two-dimensional coordinates of the target device in the video stream data.

[0112] After obtaining the staff's hand position through the above steps, it is also necessary to obtain the actual location of the customer's mobile phone before performing valet operation violation detection.

[0113] When detecting the actual location of the mobile phone, since the customer's mobile phone may be adjusted, such as being transferred from one customer's hand to another customer's hand, or to the hand of a staff member, in order to track the customer's mobile phone, it is necessary to analyze the trajectory of the second type of person, that is, the customer's target device (such as a mobile phone) to obtain the two-dimensional coordinates of the target device in the video stream data.

[0114] S16. Determine the three-dimensional coordinates of the target device according to the two-dimensional coordinates of the target device in the video stream data.

[0115] Just like the staff's hands, the two-dimensional coordinates of the customer's mobile phone also need to be converted to three-dimensional coordinates, that is, the actual position of the customer's mobile phone in three-dimensional space. The coordinate conversion process is the same as the coordinate conversion process of the staff, only the processing object is different.

[0116] S17: Determine warning information of illegal valet operation based on the three-dimensional coordinates of the hand position of the first type of person and the relative distance of the three-dimensional coordinates of the target device.

[0117] In this embodiment, after obtaining the three-dimensional coordinates corresponding to the hand position of the staff member and the three-dimensional coordinates corresponding to the position of the customer's mobile phone, the spatial distance between them can be calculated using the Euclidean distance formula, that is, the Euclidean distance between the three-dimensional coordinates of the hand position of the first type of personnel and the three-dimensional coordinates of the target device is calculated.

[0118] If the relative distance is less than the preset threshold for a period longer than the preset time and there is an action of the first type of person operating the target device, the warning information of the illegal operation on behalf of the customer is determined as an indication of the violation. Specifically, after obtaining the distance between the staff member's finger and the customer's mobile phone, when the distance is less than a certain threshold for a certain period of time and there is an obvious action of the staff member operating the customer's mobile phone, it is determined that the staff member violated the rules by operating the mobile phone on behalf of the customer. The obvious action of operating the mobile phone can be judged by combining whether the background receives the customer's instructions for handling business and whether the similarity of the front and back frames of the camera monitoring the mobile phone screen is less than a certain threshold.

[0119] If it is determined that the staff member violates the rules by operating the mobile phone on behalf of the customer, the warning information of the illegal operation on behalf of the customer is determined as an identifier representing the violation, which can be a violation text or a digital identifier, such as 1. The edge computing device will record the warning information of the illegal operation on behalf of the customer and push the warning information to the corresponding management personnel.

[0120] In this embodiment, by detecting the time when the distance is less than the threshold and whether there is any staff operating the customer's mobile phone, it is accurately judged whether there is any illegal operation behavior, thereby avoiding misjudgment caused by the large range of staff work at a certain moment, such as when the staff touches the customer's mobile phone but does not operate it, thereby improving the accuracy of illegal operation detection.

[0121] It should be noted that the various models installed in the edge computing device in this embodiment, such as the target recognition model, the personnel motion analysis model, etc., can adopt the model in the (OpenNeuralNetworkExchange, open neural network exchange format) format, which can better realize the separation of software and hardware. Adapt the model and algorithm to different edge devices and deploy them to the edge device. Obtain the camera video data stream, manually divide the recognition range according to different scenarios, and push the identified violation information to the management platform.

[0122] In this embodiment, video stream data collected by multiple camera devices is obtained, and target recognition operations are performed on the video stream data to obtain personnel detection results and object detection results; the personnel detection results include the personnel category and the position of key points of the human body of each person; the object detection results include the type and position of the object. If the position of the person's hand in the key point position of the human body of the first type of person in the personnel detection results is empty, the personnel detection results corresponding to the historical video stream are used to predict the position of the person's hand of the first type of person in the video stream data, and the three-dimensional coordinates of the position of the person's hand of the first type of person are determined according to the position of the person's hand of the first type of person corresponding to each of the video stream data. In addition, based on the object detection results corresponding to each of the video stream data, the trajectory of the target device of the second type of person is analyzed to obtain the two-dimensional coordinates of the target device in the video stream data, and the three-dimensional coordinates of the target device are determined according to the two-dimensional coordinates of the target device in the video stream data. That is, the present invention detects the three-dimensional coordinates of the staff's hand and the three-dimensional coordinates of the customer's mobile phone through the above steps, and can determine the warning information of illegal valet operation when the relative distance between the three-dimensional coordinates of the staff's hand position and the three-dimensional coordinates of the customer's mobile phone is not in compliance with the regulations, thereby realizing the detection of valet operation of mobile phones. The entire process is analyzed in real time through video streams, without the need for human participation, thus saving manpower.

[0123] In addition, in the present invention, when the hand position of the first type of personnel is not detected in the personnel detection results, the personnel detection results corresponding to the historical video stream are used to predict the hand position of the first type of personnel in the video stream data. When the hands of the staff are blocked, the hand position of the staff can be predicted, ensuring that the hand position of the staff can be obtained in real time, thereby improving the reliability and accuracy of the detection of mobile phone operation for customers.

[0124] In addition, in the present invention, based on the object detection results corresponding to each of the video stream data, the trajectory of the target device of the second type of personnel is analyzed to obtain the two-dimensional coordinates of the target device in the video stream data, and the three-dimensional coordinates of the target device in the video stream data are determined. This can monitor the movement trajectory of the user's mobile phone in real time to avoid losing the target during the handover process of the user's mobile phone, and can also improve the accuracy and reliability of the detection of the mobile phone operated on behalf of the customer.

[0125] Another embodiment of the present invention provides a specific implementation of “using the personnel detection results corresponding to the historical video stream to predict the hand positions of the first type of personnel in the video stream data”, which may include the following steps:

[0126] 1) Filter out the positions of the key points of the first type of personnel from the positions of the key points of the human body in the personnel detection results corresponding to the historical video stream.

[0127] Specifically, for the video stream data where the hand position of the staff member is not identified, when the hand is blocked for a period of time, the time information and motion trajectory can be used to predict the blocked hand position. This is a method that infers the overlapped joint points in future frames by analyzing the continuity and historical information of the motion. The core idea of ​​this method is that human body movements are usually continuous in the time dimension, especially the movements of the arms and hands will not suddenly shift greatly. Even if the hand is blocked in the first frame, the direction and position of the hand movement can be reasonably inferred through the corresponding known information of the previous frames, combined with key points such as the shoulder, elbow, and wrist motion trajectory.

[0128] Based on the personnel detection results corresponding to the historical video stream data collected by the camera device, in the frames where the hands are not blocked, the positions of the 16 key points of the staff in each frame are accurately located, and the record information of the key points in the time dimension is established.

[0129] 2) Based on the logical relationship between the non-hand key point positions and the hand key point positions, the positions of the key points of the first type of personnel are used to predict the hand positions of the first type of personnel in the video stream data.

[0130] In this embodiment, the logical relationship between the non-hand key point positions and the hand key point positions can be determined using a time series model, that is, a model for predicting the hand key point positions based on the non-hand key point positions is pre-trained, and the model is internally configured with the logical relationship between the non-hand key point positions and the hand key point positions. Then, the positions of the key points of the first type of personnel in the historical video stream are used to predict the hand positions of the staff in the current video stream data.

[0131] Among them, time series modeling is the basis for analyzing the changes in hand movements in consecutive frames. The key point position information of the past few frames can be combined as a time series, and the position of the next frame can be inferred through the long short-term memory network (LSTM). First, the required LSTM network needs to be trained, as follows:

[0132] The data prepared for training include:

[0133] Time series data of key points of the human body, especially the changes in the positions of key points over time (coordinate sequences of shoulders, elbows, wrists, and fingers).

[0134] Input data format: The key point data of each frame is expressed in two-dimensional coordinates, so the data of a single frame can be expressed as a set of two-dimensional coordinates of each key point.

[0135] The structure of the LSTM model used in training is:

[0136] The structure of LSTM consists of multiple "memory units" that can dynamically remember or forget information in the time series based on the input data. The flow of information is controlled by input, forget, and output gates.

[0137] 1. Input layer: Pass the input time series key point data into the LSTM network.

[0138] Each input data point is a multi-dimensional vector (such as joint coordinates of a frame) and is input continuously on the time axis.

[0139] 2.LSTM layer:

[0140] Input gate: determines the importance of the current input and whether to pass the key point position of the current frame to the memory unit.

[0141] Forget gate: decides whether to forget certain past information. For example, if the hand movement changes, LSTM can choose to "forget" the hand position from a long time ago.

[0142] Output gate: Based on the current input and past memory, decide what to output, that is, predict the position of the hand in the next frame.

[0143] The LSTM network extracts the law of hand movement from the input time series through these gating mechanisms. It remembers the movement patterns of the past few frames and infers the movement trends of future frames through this temporal dependency.

[0144] 3. Fully connected layer: The output of LSTM is passed to the fully connected layer, which is used to convert the hidden state of LSTM into the predicted key point position. The output of this layer is the key point coordinates of the next frame.

[0145] In the LSTM model, mean-square error (MSE) is used as the loss function to measure the difference between the predicted key point positions and the actual key point positions. In addition, the Adam optimizer is used to minimize the loss function to optimize the model.

[0146] After preparing the data and building the model structure, the LSTM model is trained. Specifically, the above time series, i.e., the positions of 16 key points in a series of continuous frames, is used as the input of the LSTM to train the model.

[0147] Subsequently, the LSTM model can be combined with input data to infer the hand position model in future frames. For example, the 16 key point positions of the previous frames in the video stream data are used as input to obtain the hand position in the current frame, and the hand position information in the occluded frame can be predicted. In addition, when predicting, the non-hand key point positions of the first type of people detected in the current video stream can also be used as input to achieve accurate prediction.

[0148] In this embodiment, when the hand position of the staff member is blocked, the hand position can be predicted, which can provide sufficient data for subsequent illegal operations and improve the accuracy of illegal operation detection.

[0149] After obtaining the hand positions of the first type of personnel corresponding to each of the video stream data, the three-dimensional coordinates of the hand positions of the first type of personnel can be determined according to the hand positions of the first type of personnel corresponding to each of the video stream data, specifically including:

[0150] 1) Calculating the parallax data of the hand positions of the first type of persons in different video stream data.

[0151] Specifically, due to the limitations of a single camera range, angle and other factors, it is difficult to determine the distance between the staff member's finger and the customer's mobile phone. The embodiment of the present invention designs a multi-camera three-dimensional distance detection algorithm, which uses a multi-camera linkage method to combine the relative positions of the cameras and the captured images to derive the three-dimensional coordinates of the object and calculate the distance between the finger and the mobile phone.

[0152] When performing specific distance calculations, the camera position needs to be calibrated in advance. First, the position of each camera (camera external parameters) and the internal parameters of the camera (focal length, principal point, etc.) need to be known. This information can be obtained by camera calibration.

[0153] In addition, it is also necessary to know the position of the detected object, which specifically refers to the position of the detected object in the viewing angle of each camera device, that is, their pixel coordinates, which specifically refers to two-dimensional coordinates. Taking the hand as an example, the position is the hand position of the staff member detected above.

[0154] After the camera device data and the hand position of the staff member have been calibrated, the three-dimensional coordinates of the hand can be determined using a three-dimensional distance detection algorithm.

[0155] First, the parallax data of the hand position of the first type of personnel in different video stream data is calculated. When calculating the parallax data, video stream data collected by two cameras are required. In the embodiment of the present invention, since a multi-channel camera device is configured, two video stream data can be randomly or according to certain rules selected from the video stream data collected by all the cameras, and the parallax is calculated using the two video stream data, and the three-dimensional coordinates of the hand are subsequently obtained using the parallax. Alternatively, the parallax is calculated once for every two video stream data, and then the corresponding three-dimensional coordinates of the hand are obtained through the parallax, and then the average value of all the three-dimensional coordinates of the hand is calculated.

[0156] When calculating parallax, for cameras with overlapping fields of view, when observing the same object from different camera perspectives, the projection position of the object will be different. Parallax refers to the difference in the projection position.

[0157] 2) Calculating the relative distance between the hand of the first type of person and the corresponding camera device based on the parallax data.

[0158] Specifically, triangulation is used to derive depth information.

[0159] Specifically, according to the parallax formula, the depth information of an object, such as a worker's hand, from the camera device (i.e., the distance from the object to the camera device) can be calculated. The specific calculation formula is as follows:

[0160] Z=f*B / d

[0161] Among them, Z is the distance from the object to the camera (depth information), f is the focal length of the camera, B is the distance between the two cameras, and d is the parallax, that is, the pixel difference between the projection position of the object in the images of the two cameras.

[0162] Substituting the parallax into the formula Z=f*B / d, the distance from the staff member's hand to the camera device can be calculated.

[0163] 3) Using the relative distance, calculate the three-dimensional coordinates of the hand position of the first type of person.

[0164] Specifically, by using relative distance and a three-dimensional distance detection algorithm, the three-dimensional coordinates of the hand positions of the first type of personnel can be obtained, that is, the positions of the personnel's hands in the actual three-dimensional space are obtained.

[0165] In this embodiment, the three-dimensional spatial coordinates of each object can be derived based on the pixel coordinates of the object under each camera device combined with triangulation, thereby obtaining the position of the staff member's hand in the actual three-dimensional space for subsequent violation judgment.

[0166] Another embodiment of the present invention provides a specific implementation of "analyzing the trajectory of the target device of the second type of person based on the object detection results corresponding to each of the video stream data to obtain the two-dimensional coordinates of the target device in the video stream data", referring to Figure 5 , which may include:

[0167] S21. Take the target device of the second type of person as a target to be tracked.

[0168] Specifically, due to the limited monitoring range of each monitoring device, obstruction by objects, insufficient field of view of the camera, etc., as people move or mobile phones are handed over, a customer's mobile phone may appear in the video stream of one camera device, but the next moment, the mobile phone disappears in the video stream and appears in the video stream of another camera device. Therefore, in order to reduce target loss, the present invention designs a multi-target tracking algorithm with multi-camera linkage. The method focuses on the association between targets among multiple cameras, especially when a target appears in one camera and then disappears and appears in another camera. These targets are regarded as the same individual, and continuous tracking of customer mobile phones is achieved to determine which mobile phone appearing in the video stream is the customer's mobile phone that needs to be monitored.

[0169] In specific implementation, the second type of person, that is, the target device (such as a mobile phone) of the customer is taken as the target to be tracked.

[0170] In actual scenarios, when determining the target to be tracked, the second type of personnel can be screened out from the personnel detection results corresponding to the video stream data, that is, the customers to be monitored can be screened out. Furthermore, the device that is closest to the second type of personnel when the device first appears is taken as the target device, and the target device is taken as the target to be tracked.

[0171] Specifically, in general, the mobile phone may be held in the customer's hand or in a bag. When the mobile phone is recognized, the distance between the center point of the mobile phone and the center point of each person is calculated in real time based on the video stream collected by the camera device. The person closest to the mobile phone is identified as the owner of the mobile phone. This information is maintained in a mapping table. Multiple cameras jointly maintain a mapping table, which stores the corresponding relationship between people and mobile phones. For example, mobile phone A belongs to person XX, and mobile phone B is input to person YY.

[0172] In this way, the mobile phone of the customer in the above-mentioned personnel detection result can be determined, and then the mobile phone is used as a target to be tracked and a target continuous tracking operation is performed.

[0173] S22. Determine appearance features of the object in the video stream data according to the corresponding object detection result.

[0174] Specifically, when tracking a target, the familiarity of the objects can be measured to determine which object is the customer's mobile phone. When measuring similarity, the appearance features can be extracted. For the object detection results, the object detection results include the object type and location. The object is marked with a label box at the location of the object, and there is a corresponding type label (also called a category label).

[0175] Based on the object detection results, the appearance features of the objects in the annotation box are extracted. Taking the mobile phone as an example, the appearance features can be the shape, color, size of the mobile phone case, etc.

[0176] When extracting appearance features, an appearance feature extractor can be used for feature extraction. The appearance feature extractor is a deep convolutional neural network (CNN), which is responsible for mapping each detected target image segment (bounding box) into a feature vector (usually a high-dimensional feature embedding), which is the extracted appearance feature.

[0177] S23: extracting appearance features of the target to be tracked.

[0178] Similarly, for the target to be tracked, an appearance feature extractor can also be used to extract features.

[0179] S24. Determine the trajectory of the target to be tracked based on the similarity between the appearance features of the object and the appearance features of the target to be tracked, so as to obtain the two-dimensional coordinates of the target device in the video stream data.

[0180] Specifically, feature matching and association are used during tracking. In specific implementation, feature matching and association are performed based on appearance features between different frames or across cameras. The essence of feature matching and association is to match the same target at different times or in different cameras by measuring similarity.

[0181] Calculate the similarity between the appearance features of the object in different frames or under different cameras and the appearance features of the target to be tracked. If the similarity is higher than a predetermined threshold, the two targets are regarded as the same target, that is, the currently analyzed object is the customer's mobile phone. If it is lower than the threshold, they are regarded as different targets, that is, the currently analyzed object is not the customer's mobile phone.

[0182] After feature matching, the detection results of the same target in consecutive frames or different camera views can be associated to generate the target's trajectory. Specifically, in consecutive frames, the target's appearance features and spatial positions are compared to associate the target and generate the trajectory. In a multi-camera system, the target's trajectory can be connected across multiple cameras.

[0183] When there are multiple customer phones in the scene, multiple customer phones are tracked at the same time, and the trajectories of different individuals are not confused by distinguishing the appearance characteristics of each target.

[0184] The tracking results of each customer's mobile phone, such as its position in the video stream, can be recorded in the above mapping table. For example, the customer's mobile phone appears at position a (specifically a two-dimensional coordinate) in the video captured by camera A, and is adjusted to position b in the video captured by camera B. As the position of the customer's mobile phone is constantly adjusted, its position in the mapping table is also constantly updated.

[0185] After determining the two-dimensional coordinates of the mobile phone in the video stream data, the two-dimensional coordinates are converted to obtain three-dimensional coordinates, and then based on the relative distance between the three-dimensional coordinates of the hand position of the first type of person and the three-dimensional coordinates of the target device, the warning information of illegal valet operation is determined.

[0186] In this embodiment, after the customer's mobile phone is identified, the mobile phone can be tracked in real time to avoid the handover of mobile phones, such as the problem of loss of recognition of user A's mobile phone after user A gives the mobile phone to user B. It can also avoid the problem of target loss across cameras, that is, the multi-target tracking algorithm using multi-camera linkage solves the target loss problem.

[0187] In addition, a mapping table is maintained by multiple cameras, and the relationship between people and mobile phones is maintained by using the mapping table to solve the problem of matching mobile phones and people, and the correspondence between mobile phones and people, as well as the location of mobile phones, is recorded in real time.

[0188] In summary, through the multi-target tracking algorithm of multi-camera linkage, the mapping table of the relationship between people and mobile phones maintained by multiple cameras, and the combination of multi-camera linkage, the distance calculation between the staff's fingers and the customer's mobile phone is realized. When the distance between the staff's fingers and the customer's mobile phone screen is less than a certain threshold for a certain period of time and there is an obvious action of operating the mobile phone, it is judged that the staff member violates the regulations by operating the mobile phone on behalf of the customer.

[0189] The present invention uses visual models and edge computing algorithms to automatically check illegal operations, and can check the operation of mobile phones in real time to promptly detect abnormal behaviors and illegal operations. The present invention greatly reduces the possibility of manual intervention and misjudgment, and can also count and analyze detection data to provide decision support and regulatory basis for bank management.

[0190] The present invention tracks the customer's mobile phone and can accurately identify the user's mobile phone even when the mobile phone is handed over. In addition, in the scenario where both the staff and the customer have mobile phones, the correspondence between the mobile phones and the personnel can also be accurately achieved.

[0191] A method for detecting illegal valet operations provided by an embodiment of the present application is introduced above. A device for executing the above method for detecting illegal valet operations will be introduced below.

[0192] See also Figure 6 , Figure 6 A schematic diagram of the structure of a detection device for illegal valet operation provided in an embodiment of the present application.

[0193] like Figure 6 As shown, the detection device for illegal valet operation includes:

[0194] A data acquisition module 11 is used to acquire video stream data collected by multiple camera devices;

[0195] The target recognition module 12 is used to perform a target recognition operation on the video stream data to obtain a person detection result and an object detection result; the person detection result includes the person category and the position of key points of each person; the object detection result includes the type and position of the object;

[0196] A position prediction module 13 is used to predict the hand position of the first type of personnel in the video stream data by using the personnel detection result corresponding to the historical video stream if the hand position of the first type of personnel in the human body key point position in the personnel detection result is empty;

[0197] A first coordinate determination module 14, configured to determine the three-dimensional coordinates of the hand positions of the first type of personnel according to the hand positions of the first type of personnel corresponding to each of the video stream data;

[0198] A second coordinate determination module 15, configured to analyze the trajectory of the target device of the second type of person based on the object detection results corresponding to each of the video stream data, so as to obtain the two-dimensional coordinates of the target device in the video stream data;

[0199] A third coordinate determination module 16, configured to determine the three-dimensional coordinates of the target device according to the two-dimensional coordinates of the target device in the video stream data;

[0200] The violation detection module 17 is used to determine the warning information of the illegal valet operation based on the three-dimensional coordinates of the hand position of the first type of personnel and the relative distance of the three-dimensional coordinates of the target device.

[0201] In a possible implementation, the location prediction module 13 includes:

[0202] A screening submodule, used to screen out the positions of the key points of the first type of personnel from the positions of the key points of the human body in the personnel detection results corresponding to the historical video stream;

[0203] A prediction submodule is used to predict the hand position of the first type of personnel in the video stream data based on the logical relationship between the non-hand key point position and the hand key point position and using the position of each key point of the first type of personnel.

[0204] In a possible implementation, the first coordinate determination module 14 includes:

[0205] A disparity calculation submodule, used to calculate disparity data of the hand positions of the first type of personnel in different video stream data;

[0206] a distance calculation submodule, configured to calculate the relative distance between the hand of the first type of person and the corresponding camera device according to the parallax data;

[0207] The coordinate calculation submodule is used to calculate the three-dimensional coordinates of the hand position of the first type of personnel using the relative distance.

[0208] In a possible implementation, the second coordinate determination module 15 includes:

[0209] a target determination submodule, configured to take a target device of a second type of person as a target to be tracked;

[0210] A feature determination submodule, used to determine the appearance features of the object in the video stream data according to the corresponding object detection result;

[0211] A feature extraction submodule, used to extract the appearance features of the target to be tracked;

[0212] The trajectory processing submodule is used to determine the trajectory of the target to be tracked based on the similarity between the appearance features of the object and the appearance features of the target to be tracked, so as to obtain the two-dimensional coordinates of the target device in the video stream data.

[0213] In a possible implementation, the target determination submodule is specifically used for:

[0214] A second type of person is screened out from the person detection results corresponding to the video stream data, and a device that is closest to the second type of person when the device first appears is taken as a target device, and the target device is taken as a target to be tracked.

[0215] In a possible implementation, the violation detection module 17 includes:

[0216] A distance calculation submodule, used to calculate the Euclidean distance between the three-dimensional coordinates of the hand position of the first type of person and the three-dimensional coordinates of the target device;

[0217] The violation detection submodule is used to determine that the warning information of the illegal valet operation is an identification representing a violation if the time length during which the relative distance is less than the preset threshold is greater than the preset time length and there is an act of the first type of person operating the target device.

[0218] In this embodiment, video stream data collected by multiple camera devices is obtained, and target recognition operations are performed on the video stream data to obtain personnel detection results and object detection results; the personnel detection results include the personnel category and the position of key points of the human body of each person; the object detection results include the type and position of the object. If the position of the person's hand in the key point position of the human body of the first type of person in the personnel detection results is empty, the personnel detection results corresponding to the historical video stream are used to predict the position of the person's hand of the first type of person in the video stream data, and the three-dimensional coordinates of the position of the person's hand of the first type of person are determined according to the position of the person's hand of the first type of person corresponding to each of the video stream data. In addition, based on the object detection results corresponding to each of the video stream data, the trajectory of the target device of the second type of person is analyzed to obtain the two-dimensional coordinates of the target device in the video stream data, and the three-dimensional coordinates of the target device are determined according to the two-dimensional coordinates of the target device in the video stream data. That is, the present invention detects the three-dimensional coordinates of the staff's hands and the three-dimensional coordinates of the customer's mobile phone through the above steps, and can determine the warning information of illegal valet operation when the relative distance between the three-dimensional coordinates of the staff's hand position and the three-dimensional coordinates of the customer's mobile phone is not in compliance, thereby realizing the detection of valet operation of mobile phones, and the entire process is analyzed in real time through video streams, without manual participation, saving manpower. In addition, in the present invention, when the hand position of the first type of personnel is not detected in the personnel detection results, the personnel detection results corresponding to the historical video stream are used to predict the hand position of the first type of personnel in the video stream data, and the hand position of the staff can be predicted when the staff's hand is blocked, ensuring that the staff's hand position can be obtained in real time, thereby improving the reliability and accuracy of valet operation of mobile phones. In addition, in the present invention, based on the object detection results corresponding to each of the video stream data, the trajectory of the target device of the second type of personnel is analyzed to obtain the two-dimensional coordinates of the target device in the video stream data, and the three-dimensional coordinates of the target device in the video stream data are determined. This can monitor the movement trajectory of the user's mobile phone in real time to avoid losing the target during the handover process of the user's mobile phone, and can also improve the accuracy and reliability of the detection of the mobile phone operated on behalf of the customer.

[0219] It should be noted that the working process of each module and sub-module in this embodiment, please refer to the corresponding process in the above embodiment, and the details will not be repeated here.

[0220] An embodiment of the present application also provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0221] The memory is used to store computer programs;

[0222] The processor is used to execute the computer program so that the electronic device can implement the above-mentioned method for detecting illegal valet operations.

[0223] refer to Figure 7 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic device in the embodiments of the present application may include but is not limited to fixed terminals such as mobile phones, notebook computers, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 7 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0224] like Figure 7 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0225] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0226] Also provided in an embodiment of the present application is a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any method for detecting illegal valet operations provided in an embodiment of the present application.

[0227] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the methods for detecting illegal valet operations provided in the embodiments of the present application.

[0228] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0229] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0230] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0231] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A method for detecting illegal valet operation, characterized in that: include: Acquire video stream data collected by multiple camera devices; Performing target recognition operation on the video stream data to obtain a person detection result and an object detection result; The personnel detection result includes the personnel category and the position of key points of each person; the object detection result includes the type and position of the object; If the position of the person's hand in the key point position of the first type of person in the person detection result is empty, predict the position of the person's hand of the first type of person in the video stream data by using the person detection result corresponding to the historical video stream; Determine the three-dimensional coordinates of the hand positions of the first type of personnel according to the hand positions of the first type of personnel corresponding to each of the video stream data; Based on the object detection results corresponding to each of the video stream data, analyzing the trajectory of the target device of the second type of person to obtain the two-dimensional coordinates of the target device in the video stream data; Determining the three-dimensional coordinates of the target device according to the two-dimensional coordinates of the target device in the video stream data; Based on the three-dimensional coordinates of the hand position of the first type of personnel and the relative distance of the three-dimensional coordinates of the target device, the warning information of illegal valet operation is determined.

2. The method for detecting illegal valet operation according to claim 1, characterized in that: Predicting the hand position of the first type of person in the video stream data by using the person detection result corresponding to the historical video stream includes: Filtering out the positions of the key points of the first type of people from the positions of the key points of the human body in the personnel detection results corresponding to the historical video stream; Based on the logical relationship between the non-hand key point positions and the hand key point positions, the positions of the key points of the first type of personnel are used to predict the hand positions of the first type of personnel in the video stream data.

3. The method for detecting illegal valet operation according to claim 1, characterized in that: Determining the three-dimensional coordinates of the hand positions of the first type of personnel according to the hand positions of the first type of personnel corresponding to each of the video stream data includes: Calculating disparity data of the hand positions of the first type of personnel in different video stream data; Calculating the relative distance between the hand of the first type of person and the corresponding camera device according to the parallax data; The three-dimensional coordinates of the hand position of the first type of person are calculated using the relative distance.

4. The method for detecting illegal valet operation according to claim 1, characterized in that: Based on the object detection results corresponding to each of the video stream data, analyzing the trajectory of the target device of the second type of person to obtain the two-dimensional coordinates of the target device in the video stream data, including: taking the target device of the second type of person as a target to be tracked; Determining appearance features of objects in the video stream data according to corresponding object detection results; Extracting appearance features of the target to be tracked; Based on the similarity between the appearance features of the object and the appearance features of the target to be tracked, the trajectory of the target to be tracked is determined to obtain the two-dimensional coordinates of the target device in the video stream data.

5. The method for detecting illegal valet operation according to claim 4, characterized in that: The target devices of the second type of personnel are taken as the targets to be tracked, including: Screening out a second type of person from the person detection results corresponding to the video stream data; The device that is closest to the second type of person when the device first appears is used as the target device; The target device is used as a target to be tracked.

6. The method for detecting illegal valet operation according to claim 1, characterized in that: Based on the three-dimensional coordinates of the hand position of the first type of personnel and the relative distance of the three-dimensional coordinates of the target device, determining the warning information of illegal valet operation includes: Calculating the Euclidean distance between the three-dimensional coordinates of the hand position of the first type of person and the three-dimensional coordinates of the target device; If the time during which the relative distance is less than the preset threshold is greater than the preset time and there is an act of the first type of person operating the target device, it is determined that the warning information of the illegal valet operation is an identification indicating a violation.

7. A detection device for illegal valet operation, characterized in that: include: A data acquisition module, used to acquire video stream data collected by multiple camera devices; A target recognition module is used to perform a target recognition operation on the video stream data to obtain a person detection result and an object detection result; the person detection result includes the person category and the position of key points of each person; the object detection result includes the type and position of the object; A position prediction module, used for predicting the position of the hand of the first type of person in the video stream data by using the person detection result corresponding to the historical video stream if the position of the hand of the first type of person in the key point position of the human body in the person detection result is empty; A first coordinate determination module, configured to determine the three-dimensional coordinates of the hand positions of the first type of personnel according to the hand positions of the first type of personnel corresponding to each of the video stream data; A second coordinate determination module, configured to analyze the trajectory of the target device of the second type of person based on the object detection results corresponding to each of the video stream data, so as to obtain the two-dimensional coordinates of the target device in the video stream data; A third coordinate determination module, used to determine the three-dimensional coordinates of the target device according to the two-dimensional coordinates of the target device in the video stream data; The violation detection module is used to determine the warning information of the illegal valet operation based on the three-dimensional coordinates of the hand position of the first type of personnel and the relative distance of the three-dimensional coordinates of the target device.

8. A computer program product, characterized in that It comprises computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the method for detecting illegal valet operation as claimed in any one of claims 1 to 6.

9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the method for detecting illegal valet operations as described in any one of claims 1 to 6.

10. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the method for detecting illegal valet operations as described in any one of claims 1 to 6.

Images