Operation violation detection method and related device

Abstract

The invention discloses an operation violation detection method and a related device, which can be applied to the field of artificial intelligence or finance. According to the method and the device, the personnel types can be monitored based on the video stream data, when the personnel types are the preset personnel types and the total number of the personnel is the preset number, the hand key points of the personnel in the video stream data are identified, and the action information of the personnel is determined based on the hand key points. When the personnel types are all the preset personnel types and the number is correct, monitoring the action information of the personnel, and subsequently determining the operation violation detection result of the specified service according to the action information of each personnel. According to the method, operation violation monitoring is carried out from the aspects of personnel types, the number and personnel actions, the monitoring requirements of actual business handling are met, and the monitoring accuracy and reliability are improved. In addition, the video stream data can be used for monitoring operation in real time, and the monitoring accuracy and reliability can be further improved.

Description

Technical Field

[0001] The present application relates to the field of operation detection technology, and in particular to an operation violation detection method and related devices. Background Art

[0002] Large-value cash business at bank branches is an important part of banking business. Large-value cash business usually involves cash deposit, withdrawal, exchange and other operations with relatively high amounts. For example, customers need to withdraw or deposit large amounts of cash, conduct cash transaction settlement, etc. These businesses may involve millions or even tens of millions of yuan in cash, so special security measures and internal control mechanisms are required to ensure the security and accuracy of transactions.

[0003] At present, when monitoring violations of large-value cash transactions, manual inspections are the main method used, which results in low accuracy in violation control. Summary of the invention

[0004] In view of the above problems, this application provides an operation violation detection method and related devices to achieve the purpose of improving the accuracy of violation monitoring of large-value cash transaction business. The specific scheme is as follows:

[0005] The first aspect of the present application provides an operation violation detection method, comprising:

[0006] Upon receiving a business instruction for handling a designated business, obtaining video stream data of a designated area for handling the designated business;

[0007] Performing a type recognition operation on each person in the video stream data to obtain a person type of each person in the video stream data;

[0008] If all of the personnel types are preset personnel types and the total number of all of the personnel is a preset number, identifying the key points of the hands of all of the personnel in the video stream data;

[0009] Determining three-dimensional hand gesture information of each of the persons based on the key points of the hands of each of the persons;

[0010] Determine the motion information of each of the personnel using the three-dimensional hand posture information of each of the personnel;

[0011] The operation violation detection result of the designated business is determined according to the action information of each of the personnel.

[0012] In a possible implementation, performing a type recognition operation on each person in the video stream data to obtain the person type of each person in the video stream data includes:

[0013] Performing a feature extraction operation on the video stream data to obtain a high-dimensional feature vector; the high-dimensional feature vector includes a vector corresponding to a person feature and a vector corresponding to an environmental feature;

[0014] Calling a preset personnel category recognition model to process the high-dimensional feature vector to obtain the personnel type of each person in the video stream data;

[0015] The training process of the preset personnel category recognition model includes:

[0016] Acquire a video stream sample, and select a first image sample that meets a training sample standard from the video stream sample;

[0017] Performing a data enhancement operation on the first image sample to obtain a second image sample;

[0018] Performing a data mixing operation on the second image sample and the public image sample to obtain a third image sample;

[0019] Determine a feature vector sample corresponding to the third image sample;

[0020] The feature vector samples are used to train a preset personnel category recognition model.

[0021] In a possible implementation, identifying the key points of the hands of each of the persons in the video stream data includes:

[0022] For a video frame in the video stream data, performing a hand region recognition operation on the video frame to obtain a hand region recognition result, and based on the hand region recognition result, segmenting a hand region image from the video frame;

[0023] A key point detection operation is performed on the hand area image to obtain a hand key point recognition result; wherein the hand key point recognition result includes hand key points, and the hand key points include at least wrists and finger joints.

[0024] In a possible implementation, determining the three-dimensional hand posture information of each of the persons based on the key points of the hands of each of the persons includes:

[0025] For the person, using the multi-view data in the video stream data, calculating the disparity information of the key points of the hand of the person;

[0026] The positions of the key points of the hand of the person are determined in three-dimensional space by using the parallax information of the key points of the hand of the person, so as to obtain the three-dimensional hand posture information of the person.

[0027] In a possible implementation, the three-dimensional hand posture information of each of the persons is used to determine the action information of each of the persons, including:

[0028] A motion trajectory analysis operation is performed on the three-dimensional hand posture information of the person to obtain the action information of the person.

[0029] In a possible implementation, determining the operation violation detection result of the designated business according to the action information of each of the personnel includes:

[0030] According to the action information of each of the persons, determining whether each of the persons has performed a plurality of preset actions;

[0031] If yes, determining that the operation violation detection result of the designated business is a first mark indicating no violation;

[0032] If not, it is determined that the operation violation detection result of the designated business is a second identifier indicating a violation.

[0033] A second aspect of the present application provides an operation violation detection device, comprising:

[0034] A data acquisition module, for acquiring video stream data of a designated area for handling a designated business upon receiving a business instruction for handling a designated business;

[0035] A type identification module, used to perform a type identification operation on each person in the video stream data to obtain a person type of each person in the video stream data;

[0036] A key point recognition module, used for identifying the hand key points of each of the persons in the video stream data if each of the person types is a preset person type and the total number of each of the persons is a preset number;

[0037] A hand posture determination module, used to determine the three-dimensional hand posture information of each of the persons based on the key points of the hands of each of the persons;

[0038] An action determination module, used to determine action information of each of the personnel using the three-dimensional hand posture information of each of the personnel;

[0039] The violation detection module is used to determine the operation violation detection result of the designated business according to the action information of each of the personnel.

[0040] A third aspect of the present application provides a computer program product, including computer-readable instructions, which, when executed on an electronic device, enables the electronic device to implement the above-mentioned operation violation detection method.

[0041] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0042] The memory is used to store computer programs;

[0043] The processor is used to execute the computer program so that the electronic device can implement the above-mentioned operation violation detection method.

[0044] A fifth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the above-mentioned operation violation detection method.

[0045] By means of the above technical scheme, the present application provides an operation violation detection method and related devices. In the present application, the personnel type can be monitored based on the video stream data. When the personnel types are all preset personnel types and the total number of personnel is the preset number, the hand key points of each of the personnel in the video stream data are identified, and the three-dimensional hand posture information of each of the personnel is determined based on the hand key points of each of the personnel. The three-dimensional hand posture information of each of the personnel is used to determine the action information of each of the personnel. That is, when the personnel types are all preset personnel types and the number is correct, the action information of the personnel is monitored, and the operation violation detection result of the specified business is determined based on the action information of each of the personnel. That is, the present invention monitors the operation violation from the aspects of personnel type, number and personnel action, which meets the monitoring requirements of actual business handling and improves the accuracy and reliability of monitoring. In addition, the present invention can use video stream data to monitor the operation in real time. Compared with the personnel inspection monitoring, which is prone to missed detection due to negligence, the present invention can further improve the accuracy and reliability of monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0047] Figure 1 A schematic diagram of a system architecture provided for this application;

[0048] Figure 2 A schematic diagram of a terminal structure provided for this application;

[0049] Figure 3 A schematic diagram of the structure of an edge computing device provided for this application;

[0050] Figure 4 A flowchart of an operation violation detection method provided for this application;

[0051] Figure 5 A flowchart of a method for determining a hand posture provided in this application;

[0052] Figure 6 A flowchart of a method for determining a personnel type provided for this application;

[0053] Figure 7 A structural diagram of an operation violation detection device provided in this application;

[0054] Figure 8 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0055] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0056] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0057] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0058] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0059] Large-value cash business at bank branches is an important part of banking business. Large-value cash business usually involves operations such as cash deposit, withdrawal, and exchange of high amounts of cash. For example, customers need to withdraw or deposit large amounts of cash, conduct cash transaction settlement, etc. These businesses may involve millions or even tens of millions of yuan in cash, so special security measures and internal control mechanisms are required to ensure the security and accuracy of transactions. Bank branches will specify a series of security assurance processes, among which the double-check of large-value cash is a very important link to ensure the accuracy and reliability of transactions. At present, when monitoring the double-check of large-value cash, computer vision technology can be used to judge violations by counting the number of people. In this case, when the type of personnel changes but the number of people does not change, the operation is still considered to be correct, and the monitoring accuracy is low. For example, during the monitoring process of two business personnel, one of them leaves and the other user enters. At this time, it is still considered that the two business personnel are reviewing, which does not meet the actual need for two staff members to review at the same time. In addition, manual inspections or manual monitoring can also be used. However, due to negligence in human monitoring, this method sometimes fails to detect violations, the accuracy of violation control is low, double-checking of large amounts of cash is often difficult to guarantee, and it will cost a lot of manpower costs.

[0060] To this end, the present invention adopts a video stream monitoring method to collect video streams of large-value cash two-person review scenes, determine the types and number of personnel in the video stream, and analyze the actions of the personnel to judge whether the large-value cash two-person review requires two business personnel to review at the same time, and whether the two business personnel are required to perform cash verification and stamping actions, which meets the actual business requirements for large-value cash two-person review.

[0061] In addition, in the present invention, when analyzing the actions of personnel, in order to improve accuracy, the key points of the hand are determined, and based on the key points of the hand, the three-dimensional hand posture information of the business personnel's hand in the actual three-dimensional space is determined, and then the three-dimensional hand posture information is analyzed for motion trajectory to obtain action information. That is, the present invention determines the action by analyzing the hand posture of the business personnel. Since the hand posture is different in different actions, the hand action of the business personnel can be accurately obtained by using the hand posture, thereby improving the detection accuracy of the hand action.

[0062] Specifically, the present application provides an operation violation detection method and related devices. In the present application, the personnel type can be monitored based on video stream data. When the personnel types are all preset personnel types and the total number of personnel is the preset number, the hand key points of each of the personnel in the video stream data are identified, and the three-dimensional hand posture information of each of the personnel is determined based on the hand key points of each of the personnel. The three-dimensional hand posture information of each of the personnel is used to determine the action information of each of the personnel. That is, when the personnel types are all preset personnel types and the number is correct, the action information of the personnel is monitored, and the operation violation detection result of the specified business is determined based on the action information of each of the personnel. That is, the present invention monitors the operation violation from the aspects of personnel type, number and personnel action, which meets the monitoring requirements of actual business handling and improves the accuracy and reliability of monitoring. In addition, the present invention can use video stream data to monitor operations in real time. Compared with personnel inspection monitoring, which is prone to missed detection due to negligence, the present invention can further improve the accuracy and reliability of monitoring.

[0063] It should be noted that the operation violation detection method and related device provided by the present invention can be used in the field of artificial intelligence or finance. The above is only an example and does not limit the application field of the operation violation detection method and related device provided by the present invention.

[0064] On the basis of the above, see Figure 1 , Figure 1 A schematic diagram of a system architecture is shown. The system may include a terminal 100 and an edge computing device 200. The edge computing device 200 may include one or more edge computing devices ( Figure 1 In the example of an edge computing device, the edge computing device 200 can provide the method provided in the embodiment of the present application for one or more terminals.

[0065] Among them, a video acquisition application can be installed on the terminal 100. The above application and web page can provide an interface. The terminal 100 can capture the video stream and send the captured video stream to the edge computing device 200. The edge computing device 200 can obtain the processing result based on the received video stream and return the processing result to the terminal 100.

[0066] It should be understood that in some optional implementations, the terminal 100 can also complete the action of obtaining the processing result based on the collected video stream by itself without the need for the cooperation of the edge computing device, and the embodiments of the present application are not limited.

[0067] Next describe Figure 1 The product form of the mid-terminal 100;

[0068] The terminal 100 in the embodiment of the present application can be a mobile phone, a camera device, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiment of the present application does not impose any limitation on this.

[0069] Figure 2 An optional hardware structure diagram of the terminal 100 is shown.

[0070] refer to Figure 2 As shown, the terminal 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), an earphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190 and other components. Those skilled in the art will appreciate that Figure 2 These are merely examples of terminals or multi-function devices and do not constitute limitations on the terminals or multi-function devices, which may include more or fewer components than those shown in the figures, or combinations of certain components, or different components.

[0071] The input unit 130 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the portable multifunctional device. Specifically, the input unit 130 may include a touch screen 131 (optional) and / or other input devices 132. The touch screen 131 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute the command sent by the processor 170; the touch signal at least includes the touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0072] Among them, the input device 132 can receive input data and the like.

[0073] The display unit 140 may be used to display information input by the user or provided to the user, various menus of the terminal 100, interactive interfaces, file displays, and / or playback of any multimedia file. In an embodiment of the present application, the display unit 140 may be used to display an interface for video stream acquisition, processing results, and the like.

[0074] The memory 120 can be used to store instructions and data. The memory 120 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc.; the instruction storage area can store software units such as operating systems, applications, instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; provide the processor 170 with hardware, software and data resources including management of computing and processing equipment, and support control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0075] The processor 170 is the control center of the terminal 100. It uses various interfaces and lines to connect various parts of the entire terminal 100. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it executes various functions of the terminal 100 and processes data, thereby controlling the terminal device as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 170. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 170 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0076] Among them, the memory 120 can be used to store software codes related to the video stream acquisition method, the processor 170 can execute the steps of the video stream acquisition method, and can also schedule other units (such as the above-mentioned input unit 130 and display unit 140) to achieve corresponding functions.

[0077] The radio frequency unit 110 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 170 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0078] In this embodiment of the present application, the radio frequency unit 110 can send data to the edge computing device 200.

[0079] It should be understood that the radio frequency unit 110 is optional and can be replaced by other communication interfaces, such as a network port.

[0080] The terminal 100 also includes a power supply 190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0081] The terminal 100 further includes an external interface 180 , which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal 100 to communicate with other devices, or to connect a charger to charge the terminal 100 .

[0082] Although not shown, the terminal 100 may also include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which are not described in detail here. Some or all of the methods described below may be applied in the following embodiments. Figure 2 In the terminal 100 shown.

[0083] Next describe Figure 1The product form of the edge computing device 200;

[0084] Figure 3 A schematic diagram of the structure of an edge computing device 200 is provided, such as Figure 3 As shown, the edge computing device 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate with each other through the bus 201.

[0085] The bus 201 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0086] The processor 202 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0087] The memory 204 may include a volatile memory, such as a random access memory (RAM). The memory 204 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard drive (HDD), or a solid state drive (SSD).

[0088] The memory 204 may be used to store software codes related to the operation violation detection method, and the processor 202 may execute the steps of the chip operation violation detection method, and may also schedule other units to implement corresponding functions.

[0089] It should be understood that the above-mentioned terminal 100 and edge computing device 200 can be centralized or distributed devices, and the processors in the above-mentioned terminal 100 and edge computing device 200 (such as processor 170 and processor 202) can be hardware circuits (such as application specific integrated circuit (ASIC), field-programmable gate array (FPGA), general-purpose processor, digital signal processor (DSP), microprocessor or microcontroller, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0090] The embodiment of the present application provides an operation violation detection method. The operation violation detection method of the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0091] Reference Figure 4 , Figure 4 A flow chart of an operation violation detection method provided in an embodiment of the present application is shown as follows: Figure 4 As shown, an operation violation detection method provided by an embodiment of the present application may include:

[0092] S11. When a service instruction for handling a designated service is received, video stream data of a designated area for handling the designated service is obtained.

[0093] Specifically, the designated business in the embodiment of the present invention is a large-value cash business. In actual scenarios, if a large-value cash business occurs, such as a customer needs to withdraw or deposit a large amount of cash, conduct cash transaction settlement, etc. When the large-value cash business begins, the business personnel can click on the operating device, such as the start button on the computer, and the edge computing device can receive the business instructions for handling the designated business. In addition, it is also possible to monitor through the camera, and when a customer enters the designated area for handling large-value cash business, a business instruction for handling the designated business can be generated and sent to the edge computing device. Alternatively, in the self-service number-taking machine, if someone takes a number for a large-value cash business, it is considered that a business instruction for handling the designated business has been received. The method of the present invention, which starts monitoring only after receiving a business instruction, can save resources compared to the real-time monitoring method.

[0094] However, in the present invention, a real-time monitoring method can also be adopted, that is, real-time video stream acquisition of the designated area for handling large-value cash business, as well as subsequent personnel type identification, personnel action detection operations, etc. This method can achieve comprehensive monitoring with high monitoring accuracy, but it will waste resources. Therefore, in actual scenarios, a suitable method is selected for monitoring operations according to needs.

[0095] In the embodiment of the present invention, when handling large-value cash transactions, a terminal needs to be installed in a designated area, such as a counter, a VIP room, etc., in order to realize monitoring, such as a camera device to collect the video stream of the designated area. Taking the camera device as an example, since the monitoring area of ​​the camera device is limited, in order to realize comprehensive monitoring of the designated area, at least one camera device can be installed. Different cameras have different installation areas and different monitoring ranges. The sum of the monitoring areas of all cameras is all the designated areas. Subsequently, the video streams collected by different cameras can be compared and processed to realize the fusion of video stream data collected by different cameras.

[0096] In order to improve the accuracy of the video stream data, a filtering operation may be performed on the video stream data to filter out impurities.

[0097] It should be noted that the present invention mainly monitors the operations of business personnel, and there is no need to monitor users who need to handle large-value cash transactions. Therefore, when collecting video stream data, the designated area may only be the area where the business personnel are located, and does not include the area where the customers are located, so as to avoid subsequent personnel type identification and motion detection operations on the customers, so as to save resources.

[0098] S12: Perform a type recognition operation on each person in the video stream data to obtain the person type of each person in the video stream data.

[0099] In the embodiment of the present invention, when handling large-value cash transactions, the bank generally requires multiple people to review, such as two business personnel reviewing at the same time. Therefore, it is necessary to perform a type recognition operation on each person in the video stream data to obtain the personnel type of each person in the video stream data. The personnel type can be business personnel and other personnel. Since business personnel generally wear bank-specific work clothes, business personnel and other personnel can be distinguished by the work clothes.

[0100] In order to realize the recognition of personnel types, in the embodiment of the present invention, a large model can be used. Since the large model is obtained through a large number of sample training, the accuracy of personnel recognition can be guaranteed. In addition to realizing the recognition of personnel types, the large model can also realize the recognition of the total number of personnel. Alternatively, after the large model is used to recognize the type of personnel, each person is marked with a character rectangular frame, and the number of character rectangular frames is subsequently counted to obtain the total number of personnel.

[0101] It should be noted that, in the embodiment of the present invention, since at least one camera device captures the video stream of the designated area, when analyzing the types of personnel and the total number of personnel, it is necessary to perform personnel type and personnel number analysis operations on all video streams.

[0102] S13: If all of the personnel types are preset personnel types and the total number of all of the personnel is a preset number, identify the key points of the hands of all of the personnel in the video stream data.

[0103] Specifically, since double verification is required when handling large-value cash transactions, in the present invention, the preset personnel type is set as a business personnel. It is determined whether the personnel identified in the video stream are all of the preset personnel type. If so, it is further determined whether there are a preset number (such as two) of business personnel, that is, the total number of personnel of each of the personnel is two. If so, it means that the personnel type and the total number of personnel meet the requirements, and the personnel actions are subsequently identified. During action recognition, image segmentation and three-dimensional gesture posture estimation can be combined to identify hand actions to identify the actions of hand stamping and cash verification.

[0104] In another implementation of the present invention, when identifying a person's action, the action can be determined by identifying the key points of the hand. In the present invention, identifying the key points of the hand of each person in the video stream data can include:

[0105] 1) For the video frames in the video stream data, a hand region recognition operation is performed on the video frames to obtain a hand region recognition result, and based on the hand region recognition result, a hand region image is segmented from the video frames.

[0106] The video frames processed in this step are the video frames in each video stream.

[0107] Specifically, an image segmentation algorithm may be used to identify the hand region in the rectangular frame of the person identified by the large model, so as to separate the hand region image from the background.

[0108] It should be noted that in addition to using image segmentation algorithms to identify and segment the hand area, a large model can also be trained and the video frames can be input into the large model, and the large model can then extract the hand area image from it.

[0109] 2) Performing a key point detection operation on the hand area image to obtain a hand key point recognition result.

[0110] Wherein, the hand key point recognition result includes the hand key points.

[0111] Specifically, in the segmented hand region image, a hand joint point detection algorithm is used to identify key points of the hand, where the hand key points include at least wrist and finger joints.

[0112] It should be noted that, for each video frame in the video stream used for the above-mentioned personnel type recognition, a hand key point recognition operation needs to be performed.

[0113] S14. Determine the three-dimensional hand posture information of each of the persons based on the hand key points of each of the persons.

[0114] For the hand key point results corresponding to each video frame in each video stream, the corresponding three-dimensional hand posture information determination operation is performed. Specifically, the three-dimensional stereoscopic shape of the human hand is estimated by using the position information of the hand key points in the video frame and combining the multi-view information of the camera device, so as to perform hand posture estimation.

[0115] To expand, refer to Figure 5 , step S14 may include:

[0116] S21. For the person, use the multi-view data in the video stream data to calculate the disparity information of the key points of the person's hands.

[0117] Specifically, since different cameras are installed in different areas and have different viewing angles, the imaging areas of the hands of the same business personnel are different in different cameras. Therefore, the same hand will have multi-viewing data in different video stream data. For example, in the first camera, the hand imaging is located in the middle part of the image, in the second camera, the hand imaging is located in the lower part of the image, and in the third camera, the hand imaging is located in the upper part of the image.

[0118] For multi-view data, for the same scene from multiple perspectives, stereo matching and disparity estimation techniques are used to calculate the disparity information between the same key points in the key points of the person's hands in different video frames. For example, for video frames with the acquisition time of XX.XX in each video frame, stereo matching and disparity estimation techniques are used to obtain the disparity information of the finger joints in different video frames.

[0119] S22. Using the parallax information of the key points of the person's hand, determine the position of the key points of the person's hand in the three-dimensional space, and obtain the three-dimensional hand posture information of the person.

[0120] Specifically, for each hand key point, the parallax information of the hand key points of the person, that is, the position information of the hand joint points from different perspectives, is fused based on the method of geometric relationship to determine the position of the hand key points of the person in the three-dimensional space, and then the positions of all the hand key points are combined to obtain more accurate and stable three-dimensional hand posture information. For example, the current three-dimensional hand posture information is pressing the name seal.

[0121] In this embodiment, multi-perspective data is fused and the current hand posture is obtained through multi-perspective analysis. Compared with the method of determining the hand posture by only using a video stream captured by a single camera device, the spatial dimension of multiple perspectives is taken into account, thereby improving the accuracy of hand posture determination.

[0122] S15. Determine the motion information of each of the persons using the three-dimensional hand posture information of each of the persons.

[0123] Specifically, for the video frames in the video stream, the three-dimensional hand posture information of each person in the video frame is obtained. Since the person's hands move during the action, the hand posture will change. Therefore, the motion trajectory analysis operation can be performed on the three-dimensional hand posture information of the person in different video frames in the same video stream to obtain the motion trajectory and changes of the key points of the hand, and obtain the action information of the person.

[0124] For example, if a business person is checking cash, he or she will have the gesture of counting money. Therefore, the trajectory analysis of the 3D hand gesture information in multiple video frames can be performed to determine whether it is the gesture of counting money. In addition, for the action of stamping a name stamp, there will be the action of pressing the name stamp and lifting the name stamp during the specific action. The trajectory analysis of the 3D hand gesture information in multiple video frames can be performed to determine whether there is the action of pressing the name stamp and lifting the name stamp.

[0125] In this embodiment, on the basis of obtaining the three-dimensional hand posture, different hand movements can be identified through trajectory analysis, and then it can be determined whether the business personnel have designated cash verification and stamping actions.

[0126] S16. Determine the operation violation detection result of the designated business according to the action information of each of the personnel.

[0127] Specifically, according to step S13, the personnel type and the total number of personnel have been identified and both meet the requirements. As long as the subsequent actions meet the requirements, the operation can be considered compliant. If the actions do not meet the requirements, the operation can be considered illegal.

[0128] When judging whether the action meets the requirements, it can be judged whether each of the persons has performed multiple preset actions according to the action information of each of the persons;

[0129] If so, determine that the operation violation detection result of the designated business is a first mark indicating no violation; if not, determine that the operation violation detection result of the designated business is a second mark indicating a violation.

[0130] Specifically, after it has been identified that there are two business personnel in total, the work of each business personnel is analyzed through the above steps. If both of them have the preset actions of cash verification and name stamping, the operation is considered to be compliant. At this time, the operation violation detection result of the specified business is the first mark indicating no violation, such as text indicating no violation, or a mark indicating no violation, such as 0. If both people do not perform the preset actions of cash verification and name stamping, the operation is considered to be illegal. At this time, the operation violation detection result of the specified business is the second mark indicating violation, such as text indicating violation, or a mark indicating violation, such as 1.

[0131] It should be noted that in actual scenarios, when large amounts of cash are double-checked, the operation process is as follows:

[0132] One of the business personnel first performs the cash verification operation, and the other business personnel starts the cash verification operation after the previous business personnel has verified a part of the cash or all the cash. After the cash verification is completed, the previous business personnel stamps the cash with a name seal, and the next business personnel also stamps the cash with a name seal after the cash verification is completed. Therefore, different business personnel perform cash verification and stamping at different times. Therefore, in the embodiment of the present invention, as long as different business personnel are detected to perform cash verification and stamping actions, there is no requirement for the time of performing the actions.

[0133] In addition, in the present invention, in order to be able to remind the corresponding personnel in time when the operation violates the regulations. When the business starts to be handled, the video stream can be analyzed and operated. Later, when the business personnel click the business completion button, it is judged whether the operation violation detection result obtained is not a violation. If it is not a violation, it means that the business handling process is correct. If it is a violation, it means that the business handling process is incorrect, and the early warning information is sent to the management platform. The management platform receives the early warning information, records and counts the early warning information, outputs the early warning statistical information and the early warning result push, such as sending the early warning statistical information and the early warning result to the mobile phone of the management personnel, so that the management personnel can pay attention to the operation violation event in time and make corresponding operation corrections.

[0134] In addition, the present invention can also be combined with a face recognition algorithm to determine whether to change people for review, so as to ensure that both business personnel have performed the cash verification and stamping actions.

[0135] In this embodiment, the personnel type can be monitored based on the video stream data. When the personnel types are all preset personnel types and the total number of personnel is the preset number, the key points of the hands of each of the personnel in the video stream data are identified, and the three-dimensional hand posture information of each of the personnel is determined based on the key points of the hands of each of the personnel. The action information of each of the personnel is determined using the three-dimensional hand posture information of each of the personnel. That is, when the personnel types are all preset personnel types and the number is correct, the action information of the personnel is monitored, and the operation violation detection result of the designated business is determined based on the action information of each of the personnel. That is, the present invention monitors the operation violation from the aspects of personnel type, number and personnel action, which meets the monitoring requirements of actual business handling and improves the accuracy and reliability of monitoring. In addition, the present invention can use video stream data to monitor operations in real time. Compared with the situation that personnel patrol monitoring is prone to missed detection due to negligence, the present invention can further improve the accuracy and reliability of monitoring.

[0136] An application scenario of the present invention may be as follows:

[0137] In actual scenarios, the camera video data stream is obtained and the recognition range is manually divided according to different scenarios (for example, only a certain area in the video frame is circled as the focus of subsequent analysis, and the uncircled part is not subjected to subsequent recognition processing).

[0138] When the background recognizes that a large cash transaction has begun, the information is sent to the edge computing device. The personnel category recognition module is used to determine whether the person handling the business is a business person; if it is a business person, the gesture action recognition module is used to determine whether the business person has cash verification and name stamping actions; the face detection module is used to determine whether two different business persons have cash verification and name stamping actions; if the above conditions are met, the large cash business process is compliant; if not, it is illegal, and the identified violation information is pushed to the management platform, and the specific violation information is prompted.

[0139] For the edge computing device in the embodiment of the present invention, the above-mentioned model can adopt an ONNX (Open Neural Network Exchange) format model, which can better realize the separation of software and hardware, so as to adapt the model and algorithm to different edge devices for easy deployment in the edge device.

[0140] According to the above content, the present invention mainly relates to the following modules:

[0141] Personnel category identification module: Receives labeled data and public data verified by business personnel during large cash transactions, trains a personnel category identification model, and outputs a model pt file that can identify business personnel and other personnel. The model pt file will be installed in the edge computing device.

[0142] Gesture action recognition module: receives the person position rectangle frame identified by the person category recognition module, analyzes the person's hand movements, and outputs whether there are cash verification and stamping actions.

[0143] Face detection module: receives the person position rectangle frame from the person category recognition module, analyzes the face information, and outputs the face information.

[0144] The algorithm module for judging the double-person review of large-value cash transactions: it receives the outputs of the personnel category recognition module, the gesture recognition module, and the face detection module, determines whether the process is compliant when there is a large-value cash transaction, and outputs specific compliance or violation information.

[0145] Management platform module: receive warning information, record and count warning information, output warning statistics and push warning results.

[0146] In addition, an edge computing box module is also set up: the onnx format model of the target recognition module, the personnel action analysis module, and the logical algorithm of the personnel behavior analysis algorithm module are received, adapted and deployed to the edge computing box. The camera video data stream is obtained, the recognition range is manually divided according to different scenarios, and the identified violation information is pushed to the management platform.

[0147] In this embodiment, in order to realize fully automatic supervision of the behavior of double-checking large amounts of cash, the visual model and edge computing algorithm are deployed in the edge computing box, and the function of real-time automatic warning of violations is realized through the management platform. The visual model is used to obtain personnel information, including personnel category and personnel behavior, and the personnel behavior analysis algorithm is used to determine whether there is a violation by combining these personnel information. When a violation occurs, the violation information is pushed to the management platform in a timely manner. The management platform records and counts the violations, notifies the relevant personnel in real time to handle the warning information, and realizes the visual automatic inspection of violations. That is, the present invention can fully automatically analyze and warn whether the process is compliant when handling large-amount cash business, and promptly discover abnormal behavior and illegal operations. The present invention helps to improve the accuracy and transparency of transactions, enhance internal control, reduce potential errors and fraudulent behaviors, and can also count and analyze the detection data to provide decision support and regulatory basis for bank management.

[0148] In addition, the present invention is mainly implemented on edge computing devices, and does not centrally send data to the cloud for processing, which can reduce the amount of data processing in the cloud and reduce the amount of cloud tasks.

[0149] The above embodiment mentioned that it is necessary to determine the personnel type of each person in the video stream data, and now the specific implementation thereof is introduced. Figure 6, performing a type recognition operation on each person in the video stream data to obtain the person type of each person in the video stream data may include:

[0150] S31. Perform feature extraction on the video stream data to obtain a high-dimensional feature vector.

[0151] In actual scenarios, if the video stream is directly used as the input of the model, the amount of input data is too large, and there are many interferences in the data stream, such as some interferences in the designated area that have no effect on personnel recognition, such as cabinets, etc. Therefore, in order to reduce the amount of data subsequently input to the model and improve the accuracy of model recognition, feature extraction operations are performed on the video frames in the video stream data to obtain high-dimensional feature vectors.

[0152] The high-dimensional feature vector includes a vector corresponding to personnel features and a vector corresponding to environmental features. Personnel features may include clothing, hairstyle, etc. Generally, bank staff have uniform clothing and hairstyle. Environmental features may include whether there is a working terminal, such as a computer, whether there is a device for collecting faces or fingerprints, etc. Generally, when bank staff handle business, they will handle it on a computer and will perform identity verification through a device for collecting faces or fingerprints.

[0153] When performing feature extraction, a large model can also be used to implement it. By inputting the video stream into the pre-trained large model, a high-dimensional feature vector corresponding to each video frame in the video stream can be obtained.

[0154] S32: calling a preset personnel category recognition model to process the high-dimensional feature vector to obtain the personnel type of each person in the video stream data.

[0155] Specifically, a preset personnel category recognition model may be pre-trained, and the high-dimensional feature vector may be processed using the model to obtain the personnel type of each person in the video stream data.

[0156] The model in this embodiment may be, for example, a neural network model, a machine learning model, etc. The training process of the preset personnel category recognition model may include:

[0157] 1) Obtain video stream samples, and select first image samples that meet training sample standards from the video stream samples.

[0158] Specifically, when training the model, a large number of training samples are required. The samples can come from video streams of business personnel collected when handling large cash transactions in actual scenarios.

[0159] In order to improve the accuracy and reliability of samples, training sample standards for screening images can be formulated to screen out high-priority or high-quality samples. The training sample standards can include image clarity, content diversity, whether the content is a cash business processing scene at a branch, etc.

[0160] Among them, the clarity of the image can be set to standard, high-definition, etc. The diversity of the content refers to whether the content includes two factors: people and environment. It cannot be only people or only environment. Whether the content is a cash business processing scene at a branch is to exclude samples that are not related to large cash transactions.

[0161] After obtaining the video stream sample, each video frame in the video stream can be automatically screened using the training sample standard to obtain the first image sample. During the screening, video frames that do not meet the training sample standard are eliminated, and only video frames that meet the training sample standard are retained.

[0162] In addition, in order to avoid the problem of inaccurate training caused by too many similar samples in the image samples, the similarity of each sample in the first image sample can be calculated, and only one sample with high similarity (such as greater than a preset similarity threshold) can be retained to ensure sample diversity and avoid duplicate samples affecting the training results.

[0163] 2) Performing a data enhancement operation on the first image sample to obtain a second image sample.

[0164] Specifically, in order to improve the diversity of training samples, data enhancement operations such as rotation, scaling, and translation may be performed on the first image samples to generate more sample data as a training set. The samples obtained at this time are called second image samples.

[0165] 3) Performing a data mixing operation on the second image sample and the public image sample to obtain a third image sample.

[0166] Specifically, there will be corresponding images or videos of large cash transactions on the web page. If it is an image, it will be directly used as a public image sample. If it is a video, the video frame can be used as a public image sample.

[0167] Then, the second image samples and the public image samples are mixed in a certain ratio to expand the sample size.

[0168] It should be noted that, for the public image samples, a screening operation can also be performed using the above-mentioned training sample standards, and only the public image samples that meet the training sample standards are retained, and then subsequent mixing operations are performed.

[0169] 4) Determine the feature vector sample corresponding to the third image sample.

[0170] The process of determining the feature vector sample corresponding to the third image sample is the same as the process of performing feature extraction operation on the video stream data. Please refer to the corresponding description above.

[0171] 5) Using the feature vector samples, a preset personnel category recognition model is trained.

[0172] Before training the model, the third image sample needs to be labeled, which can be to mark the personnel in the third image sample with a rectangular frame and label the personnel type, such as business personnel and other personnel. Then, the model is trained using the feature vector sample and the personnel type label corresponding to the third image sample. When the number of training times reaches the set value or the loss function meets the requirements, the training is stopped to obtain the preset personnel category recognition model and put it into use.

[0173] During actual use, the video stream or image is input into the preset personnel category recognition model, and the preset personnel category recognition model can identify whether the person in the video stream or image is a business person or other person, and the model will frame the recognized target with a rectangular frame.

[0174] In this embodiment, during model training, samples that meet the training sample standards are screened to ensure the quality of the samples, thereby improving the reliability of model training. In addition, by means of data enhancement and mixing with public data, the diversity of samples is expanded to further improve the reliability of model training.

[0175] An operation violation detection method provided in an embodiment of the present application is introduced above, and a device for executing the above operation violation detection method will be introduced below.

[0176] See also Figure 7 , Figure 7 This is a schematic diagram of the structure of an operation violation detection device provided in an embodiment of the present application. Figure 7 As shown, the operation violation detection device comprises:

[0177] The data acquisition module 11 is used to acquire the video stream data of the designated area for handling the designated business when receiving the business instruction for handling the designated business;

[0178] A type identification module 12, configured to perform a type identification operation on each person in the video stream data to obtain a person type of each person in the video stream data;

[0179] A key point identification module 13, for identifying a hand key point of each of the persons in the video stream data if each of the person types is a preset person type and the total number of each of the persons is a preset number;

[0180] A hand posture determination module 14, used to determine the three-dimensional hand posture information of each of the persons based on the key points of the hands of each of the persons;

[0181] An action determination module 15, used to determine the action information of each of the personnel using the three-dimensional hand posture information of each of the personnel;

[0182] The violation detection module 16 is used to determine the operation violation detection result of the designated business according to the action information of each of the personnel.

[0183] In a possible implementation, the type identification module 12 includes:

[0184] A feature extraction submodule, used to perform feature extraction operation on the video stream data to obtain a high-dimensional feature vector; the high-dimensional feature vector includes a vector corresponding to a person feature and a vector corresponding to an environmental feature;

[0185] A type identification submodule, used to call a preset personnel category identification model, process the high-dimensional feature vector, and obtain the personnel type of each person in the video stream data;

[0186] The training process of the preset personnel category recognition model includes:

[0187] Acquire a video stream sample, and select a first image sample that meets a training sample standard from the video stream sample;

[0188] Performing a data enhancement operation on the first image sample to obtain a second image sample;

[0189] Performing a data mixing operation on the second image sample and the public image sample to obtain a third image sample;

[0190] Determine a feature vector sample corresponding to the third image sample;

[0191] The feature vector samples are used to train a preset personnel category recognition model.

[0192] In a possible implementation, the key point identification module 13 includes:

[0193] a segmentation submodule, for performing a hand region recognition operation on a video frame in the video stream data to obtain a hand region recognition result, and segmenting a hand region image from the video frame based on the hand region recognition result;

[0194] The key point recognition submodule is used to perform a key point detection operation on the hand area image to obtain a hand key point recognition result; wherein the hand key point recognition result includes hand key points, and the hand key points include at least wrists and finger joints.

[0195] In a possible implementation, the hand posture determination module 14 includes:

[0196] A disparity determination submodule, for calculating disparity information of key points of the hand of the person using multi-view data in the video stream data;

[0197] The gesture recognition submodule is used to determine the position of the key points of the hand of the person in the three-dimensional space by using the parallax information of the key points of the hand of the person, and obtain the three-dimensional hand gesture information of the person.

[0198] In a possible implementation, the action determination module 15 is specifically configured to:

[0199] A motion trajectory analysis operation is performed on the three-dimensional hand posture information of the person to obtain the action information of the person.

[0200] In a possible implementation, the violation detection module 16 is specifically used to:

[0201] Based on the action information of each of the personnel, determine whether each of the personnel has performed multiple preset actions; if so, determine that the operation violation detection result of the designated business is a first mark indicating no violation; if not, determine that the operation violation detection result of the designated business is a second mark indicating a violation.

[0202] In this embodiment, the personnel type can be monitored based on the video stream data. When the personnel types are all preset personnel types and the total number of personnel is the preset number, the key points of the hands of each of the personnel in the video stream data are identified, and the three-dimensional hand posture information of each of the personnel is determined based on the key points of the hands of each of the personnel. The action information of each of the personnel is determined using the three-dimensional hand posture information of each of the personnel. That is, when the personnel types are all preset personnel types and the number is correct, the action information of the personnel is monitored, and the operation violation detection result of the designated business is determined based on the action information of each of the personnel. That is, the present invention monitors the operation violation from the aspects of personnel type, number and personnel action, which meets the monitoring requirements of actual business handling and improves the accuracy and reliability of monitoring. In addition, the present invention can use video stream data to monitor operations in real time. Compared with the situation that personnel patrol monitoring is prone to missed detection due to negligence, the present invention can further improve the accuracy and reliability of monitoring.

[0203] It should be noted that, for the working process of each module and sub-module in the embodiment of the present invention, please refer to the corresponding description in the above embodiment, which will not be repeated here.

[0204] Another embodiment of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0205] The memory is used to store computer programs;

[0206] The processor is used to execute the computer program so that the electronic device can implement the above-mentioned operation violation detection method.

[0207] refer to Figure 8 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic device in the embodiments of the present application may include but is not limited to fixed terminals such as mobile phones, notebook computers, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 8 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0208] like Figure 8 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0209] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 8 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0210] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the operation violation detection methods provided in the embodiments of the present application.

[0211] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any operation violation detection method provided in the embodiment of the present application.

[0212] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0213] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0214] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0215] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A method for detecting operation violations, characterized in that: include: Upon receiving a business instruction for handling a designated business, obtaining video stream data of a designated area for handling the designated business; Performing a type recognition operation on each person in the video stream data to obtain a person type of each person in the video stream data; If all of the personnel types are preset personnel types and the total number of all of the personnel is a preset number, identifying the key points of the hands of all of the personnel in the video stream data; Determining three-dimensional hand gesture information of each of the persons based on the key points of the hands of each of the persons; Determine the motion information of each of the personnel using the three-dimensional hand posture information of each of the personnel; The operation violation detection result of the designated business is determined according to the action information of each of the personnel.

2. The operation violation detection method according to claim 1, characterized in that: Performing a type identification operation on each person in the video stream data to obtain the person type of each person in the video stream data includes: Performing a feature extraction operation on the video stream data to obtain a high-dimensional feature vector; the high-dimensional feature vector includes a vector corresponding to a person feature and a vector corresponding to an environmental feature; Calling a preset personnel category recognition model to process the high-dimensional feature vector to obtain the personnel type of each person in the video stream data; The training process of the preset personnel category recognition model includes: Acquire a video stream sample, and select a first image sample that meets a training sample standard from the video stream sample; Performing a data enhancement operation on the first image sample to obtain a second image sample; Performing a data mixing operation on the second image sample and the public image sample to obtain a third image sample; Determine a feature vector sample corresponding to the third image sample; The feature vector samples are used to train a preset personnel category recognition model.

3. The operation violation detection method according to claim 1, characterized in that: Identifying key points of hands of each of the persons in the video stream data includes: For a video frame in the video stream data, performing a hand region recognition operation on the video frame to obtain a hand region recognition result, and based on the hand region recognition result, segmenting a hand region image from the video frame; A key point detection operation is performed on the hand area image to obtain a hand key point recognition result; wherein the hand key point recognition result includes hand key points, and the hand key points include at least wrists and finger joints.

4. The operation violation detection method according to claim 1, characterized in that: Based on the key points of the hands of each of the personnel, determining the three-dimensional hand posture information of each of the personnel includes: For the person, using the multi-view data in the video stream data, calculating the disparity information of the key points of the hand of the person; The positions of the key points of the hand of the person are determined in three-dimensional space by using the parallax information of the key points of the hand of the person, so as to obtain the three-dimensional hand posture information of the person.

5. The operation violation detection method according to claim 1, characterized in that: Using the three-dimensional hand posture information of each of the personnel, the action information of each of the personnel is determined, including: A motion trajectory analysis operation is performed on the three-dimensional hand posture information of the person to obtain the action information of the person.

6. The operation violation detection method according to claim 1, characterized in that: Determine the operation violation detection result of the designated business according to the action information of each of the personnel, including: According to the action information of each of the persons, determining whether each of the persons has performed a plurality of preset actions; If yes, determining that the operation violation detection result of the designated business is a first mark indicating no violation; If not, it is determined that the operation violation detection result of the designated business is a second identifier indicating a violation.

7. An operation violation detection device, characterized in that: include: A data acquisition module, for acquiring video stream data of a designated area for handling a designated business upon receiving a business instruction for handling a designated business; A type identification module, used to perform a type identification operation on each person in the video stream data to obtain a person type of each person in the video stream data; A key point recognition module, used for identifying the hand key points of each of the persons in the video stream data if each of the person types is a preset person type and the total number of each of the persons is a preset number; A hand posture determination module, used to determine the three-dimensional hand posture information of each of the persons based on the key points of the hands of each of the persons; An action determination module, used to determine action information of each of the personnel using the three-dimensional hand posture information of each of the personnel; The violation detection module is used to determine the operation violation detection result of the designated business according to the action information of each of the personnel.

8. A computer program product, characterized in that It comprises computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the operation violation detection method as claimed in any one of claims 1 to 6.

9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the operation violation detection method as described in any one of claims 1 to 6.

10. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the operation violation detection method as described in any one of claims 1 to 6.

Images