Vault access behavior monitoring method and related device

Abstract

The invention discloses a monitoring method for a vault access behavior and a related device, which can be applied to the field of artificial intelligence or finance. And performing vault door identification and personnel identification on the monitoring image by using a pre-deployed vault door identification model and personnel identification model, constructing a two-dimensional coordinate system in a vault door plane based on the target monitoring image, and obtaining a vault door two-dimensional coordinate set and two-dimensional personnel position information after two-dimensional mapping, based on the two-dimensional coordinate set of the vault door and the two-dimensional personnel position information, generating an action judgment result, based on the personnel identification result of each personnel monitoring image and the action judgment result sequence, judging whether a pre-configured vault warehouse-in and warehouse-out rule is met, and giving out an early warning when the rule is not met. According to the scheme, the accuracy of action judgment is improved through two-dimensional mapping, and the real-time performance and accuracy of vault access behavior monitoring are achieved through the edge side terminal.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a monitoring method and related device for entering and exiting a safe. Background Art

[0002] The vault is a very important security location, which is distributed all over the country and involves many high-standard compliance requirements, regulatory requirements and internal control requirements. As a key security area in financial scenarios, the vault is equipped with multiple cameras. Taking the bank vault as an example, the bank security system uses multiple monitors to perform round-the-clock monitoring of the environment in each area of ​​the vault. For different vault areas, the bank security system monitors different objects. The focus of monitoring the vault door area is to monitor whether the behavior of personnel entering and leaving the vault meets the entry and exit specifications. For example, whether the type of personnel entering the vault, the number of personnel, and the time when each person enters the vault meet the entry and exit specifications.

[0003] At present, the entry and exit behaviors of personnel are checked in real time by manually observing the monitoring images, or by random inspection. Both methods rely on manual observation of the monitoring images, and after discovering the possible risks, the alarm device of the security system is manually triggered. Manual observation of the monitoring images may result in omissions or misdetections, and thus the bank security system cannot guarantee the real-time and accuracy of the risk alarm for the entry and exit behaviors of personnel. Summary of the invention

[0004] In view of the above problems, the present application provides a method and related device for monitoring the in and out behavior of the vault, so as to achieve the purpose of improving the real-time and accuracy of the monitoring of the in and out behavior of the vault. The specific scheme is as follows:

[0005] The first aspect of the present application provides a method for monitoring in-and-out vault behavior, which is applied to a terminal and includes:

[0006] The monitoring images in the real-time monitoring image sequence are sequentially input into the safe door recognition model to obtain the safe door recognition result output by the safe door recognition model; the safe door recognition result includes the safe door opening and closing state and the safe door key point coordinate set;

[0007] If a change in the switch state of the safe door is detected, starting from the target monitoring image, the monitoring images in the real-time monitoring image sequence are sequentially input into the personnel recognition model to obtain the personnel recognition result output by the personnel recognition model, the personnel recognition result includes the personnel type and personnel location information, and the target monitoring image is a monitoring image of the change in the switch state of the safe door; wherein the safe door recognition model and the personnel recognition model are both obtained by the server training a machine learning model based on a training data set;

[0008] Based on the target monitoring image, construct a two-dimensional coordinate system in the safe door plane as a two-dimensional safe door coordinate system;

[0009] Mapping the safe door key point coordinate set of the target monitoring image to the two-dimensional safe door coordinate system to obtain a two-dimensional safe door coordinate set;

[0010] For each personnel monitoring image, mapping the personnel position information of the personnel monitoring image to the two-dimensional safe door coordinate system to obtain the two-dimensional personnel position information, wherein the personnel monitoring image is the monitoring image input into the personnel recognition model;

[0011] For each of the personnel monitoring images, based on the two-dimensional coordinate set of the safe door and the two-dimensional personnel position information, an action judgment result is generated, wherein the action judgment result indicates whether the target person in the personnel monitoring image triggers an action of entering or exiting the safe door;

[0012] Based on the personnel recognition results and action determination result sequences of each of the personnel monitoring images, judging whether the pre-configured vault entry and exit rules are met, the action determination result sequence comprising the action determination results of the personnel monitoring images arranged in chronological order;

[0013] If the vault entry and exit rules are not met, an early warning message will be issued.

[0014] Optionally, the method for monitoring the in and out behavior of the vault also includes:

[0015] After obtaining the identification results of each safe door output by the safe door identification model, updating the safe door switch state sequence, wherein the safe door switch state sequence includes the safe door switch states of each monitoring image arranged in chronological order;

[0016] Determine whether the safe door switch state changes based on the safe door switch state sequence.

[0017] Optionally, based on the target monitoring image, a two-dimensional coordinate system in the safe door plane is constructed as a two-dimensional safe door coordinate system, including:

[0018] Based on the safe door key point coordinate set, generate a safe door vertex coordinate set;

[0019] A two-dimensional coordinate system in the vault door plane is established with a horizontal line passing through the lowest vertex in the vault door vertex coordinate set and a vertical line passing through the leftmost vertex in the vault door vertex coordinate set as coordinate axes as the two-dimensional vault door coordinate system;

[0020] The key point coordinate set of the safe door of the target monitoring image is mapped to the two-dimensional safe door coordinate system to obtain the two-dimensional coordinate set of the safe door, including:

[0021] The coordinates of each vertex in the vault door vertex coordinate set are mapped to the two-dimensional vault door coordinate system to obtain the two-dimensional vault door coordinate set.

[0022] Optionally, mapping the personnel position information of the personnel monitoring image to the two-dimensional safe door coordinate system to obtain the two-dimensional personnel position information includes:

[0023] Acquire a side coordinate set of a person rectangular frame based on the person position information, wherein the side coordinate set includes a longitudinal coordinate of each horizontal side and a transverse coordinate of each vertical side of the person rectangular frame;

[0024] The edge coordinate set of the personnel rectangular frame is mapped to the two-dimensional safe door coordinate system to obtain the two-dimensional personnel position information.

[0025] Optionally, based on the safe door two-dimensional coordinate set and the two-dimensional personnel position information, generating an action judgment result includes:

[0026] Determine whether the rectangle formed by each coordinate in the two-dimensional personnel position information is within the outline formed by each vertex in the two-dimensional coordinate set of the safe door;

[0027] If so, it is determined that the person triggers the action of entering or exiting the safe door.

[0028] Optionally, the method for monitoring the in-and-out behavior of the vault further includes:

[0029] Loading the safe door recognition model and the personnel recognition model, and setting the initial states of the safe door recognition model and the personnel recognition model to closed;

[0030] If a change in the switch state of the safe door is detected, the safe door recognition model is turned off and the personnel recognition model is turned on;

[0031] When the preset time threshold is reached or the rule determination result is non-compliance, the personnel recognition model is closed and the safe door recognition model is opened.

[0032] The second aspect of the present application provides a monitoring device for entering and exiting a safe, comprising:

[0033] A safe door recognition unit is used to sequentially input the monitoring images in the real-time monitoring image sequence into the safe door recognition model to obtain the safe door recognition result output by the safe door recognition model; the safe door recognition result includes the safe door opening and closing state and the safe door key point coordinate set;

[0034] A personnel identification unit is used to input the monitoring images in the real-time monitoring image sequence into the personnel identification model in sequence, starting from the target monitoring image, to obtain the personnel identification result output by the personnel identification model, wherein the personnel identification result includes the personnel type and personnel location information, and the target monitoring image is a monitoring image in which the door opening and closing state of the safe door changes; wherein the safe door identification model and the personnel identification model are both obtained by training a machine learning model based on a training data set by the server;

[0035] A two-dimensional coordinate system construction unit, used to construct a two-dimensional coordinate system in the safe door plane based on the target monitoring image as a two-dimensional safe door coordinate system;

[0036] A two-dimensional coordinate mapping unit is used to map the safe door key point coordinate set of the target monitoring image to the two-dimensional safe door coordinate system to obtain the safe door two-dimensional coordinate set; for each personnel monitoring image, the personnel position information of the personnel monitoring image is mapped to the two-dimensional safe door coordinate system to obtain the two-dimensional personnel position information, and the personnel monitoring image is the monitoring image input into the personnel recognition model;

[0037] An action judgment unit, for generating an action judgment result for each of the personnel monitoring images based on the two-dimensional coordinate set of the safe door and the two-dimensional personnel position information, wherein the action judgment result indicates whether the target person in the personnel monitoring image triggers an action of entering or exiting the safe door;

[0038] A rule judgment unit, used to judge whether the pre-configured vault entry and exit rules are met based on the personnel recognition results and action judgment result sequences of each of the personnel monitoring images, wherein the action judgment result sequences include the action judgment results of the personnel monitoring images arranged in chronological order;

[0039] The early warning unit is used to issue an early warning message if the vault entry and exit rules are not met.

[0040] The third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the method for monitoring the entry and exit behavior of the safe according to the first aspect or any implementation method of the first aspect.

[0041] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0042] The memory is used to store computer programs;

[0043] The processor is used to execute the computer program so that the electronic device can implement the method for monitoring the entry and exit behavior of the safe according to the first aspect or any implementation method of the first aspect.

[0044] A fifth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can perform a method for monitoring the entry and exit behavior of the safe according to the first aspect or any implementation method of the first aspect.

[0045] By means of the above technical scheme, the present application provides a monitoring method and related device for entering and exiting the safe, wherein the monitoring images in the real-time monitoring image sequence are sequentially input into the safe door recognition model to obtain the safe door recognition result output by the safe door recognition model. If it is detected that the switch state of the safe door changes, starting from the target monitoring image, the monitoring images in the real-time monitoring image sequence are sequentially input into the personnel recognition model to obtain the personnel recognition result output by the personnel recognition model, and the target monitoring image is the monitoring image of the switch state of the safe door changes. Based on the target monitoring image, a two-dimensional coordinate system in the safe door plane is constructed as a two-dimensional safe door coordinate system. The safe door key point coordinate set of the target monitoring image is mapped to the two-dimensional safe door coordinate system to obtain the safe door two-dimensional coordinate set. For each personnel monitoring image, the personnel position information of the personnel monitoring image is mapped to the two-dimensional safe door coordinate system to obtain the two-dimensional personnel position information, and the personnel monitoring image is the monitoring image input into the personnel recognition model. For each personnel monitoring image, an action judgment result is generated based on the safe door two-dimensional coordinate set and the two-dimensional personnel position information. Based on the personnel recognition results and action judgment result sequence of each personnel monitoring image, it is judged whether it complies with the pre-configured vault entry and exit rules. The action judgment result sequence includes the action judgment results of the personnel monitoring images arranged in chronological order. If it does not comply with the vault entry and exit rules, an early warning message is issued. It can be seen that this solution improves the real-time and accuracy of the vault entry and exit behavior monitoring through the terminal using the pre-deployed vault door recognition model and personnel recognition model, and through the two-dimensional vault door coordinate system, the personnel location information and the coordinates of the key points of the vault door are mapped to the two-dimensional plane, which reduces the difficulty of action judgment and improves the accuracy of the action judgment results, thereby further improving the accuracy of the vault entry and exit behavior monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0047] Figure 1 An architectural diagram of a distributed monitoring system for entering and exiting a vault provided in this application;

[0048] Figure 2A schematic diagram of the hardware structure of a terminal provided in this application;

[0049] Figure 3 A schematic diagram of the hardware structure of a server provided for this application;

[0050] Figure 4 A schematic diagram of the specific structure of a monitoring system for entering and exiting a safe provided in this application;

[0051] Figure 5 A flowchart of a specific implementation of a method for monitoring in and out of a vault provided in an embodiment of the present application;

[0052] Figure 6 A flowchart of a method for monitoring entry and exit behavior provided in an embodiment of the present application;

[0053] Figure 7 A schematic diagram of the structure of a monitoring device for entering and exiting a safe provided in an embodiment of the present application;

[0054] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0056] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0057] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0058] The present application can be applied in the technical field of risk monitoring of bank vaults, and specifically can be applied in the scenario of monitoring the behaviors of entering and leaving the vault.

[0059] Banks have a set of specifications for the types and numbers of people entering the vault, as well as the door opening and closing procedures, including that specific types of personnel must perform vault operations according to specified requirements every day. The traditional inspection method is to manually observe surveillance images for inspection, find violations and manually trigger alarms. This method has the disadvantages of large workload, high labor costs, easy omissions and low coverage. Therefore, in actual operations, sampling inspections are often used to find violations and manually trigger alarms. Limited by the subjectivity of manual observation and differences in ability, the existing bank security system has technical shortcomings such as low real-time performance, low accuracy, and high lag in risk monitoring of in-and-out behaviors.

[0060] In order to solve the above technical shortcomings, this solution proposes to collect monitoring images of the area inside the safe door in real time based on machine vision technology, and use the pre-trained safe door recognition model and personnel recognition model to obtain the safe door status information and personnel status information respectively, and judge the entry and exit behavior information corresponding to the monitoring image based on the safe door status information and personnel status information, and judge whether it complies with the entry and exit behavior norms based on the entry and exit behavior information of multiple monitoring images within a preset time period. If it does not comply, it is determined that there is a risk of entry and exit behavior, and the bank security system automatically triggers the risk of entry and exit behavior. The alarm device of the bank security system is triggered. Among them, the safe door recognition model and the personnel recognition model are constructed and optimized based on the deep learning model, and the entry and exit behavior norms are pre-configured, specifically including: personnel type rules, personnel quantity rules, and entry and exit time rules.

[0061] Thus, the monitoring image is used to monitor the state of the vault door and the behavior of personnel through the recognition model (including the vault door recognition model and the personnel recognition model), and the in-and-out behavior information is obtained based on the recognized state of the vault door and the behavior of personnel, thereby realizing the risk monitoring of in-and-out vault behavior. This solution can significantly reduce the risk and labor cost of manual supervision and improve the real-time monitoring of in-and-out vault behavior. Furthermore, this solution can solve the problems of large manual workload and delayed inspection, and can cover all vault environments and all time periods, reducing omissions and subjective errors in manual inspections.

[0062] Research has found that the bank security system is deployed in the form of a central server, and the deep learning model is deployed on the server in the bank security system (such as a cloud server). The image sensitive information involved in the in-and-out behavior involves security and privacy protection issues during the transmission to the server, which may lead to the leakage of image sensitive information or malicious use. In this regard, this solution further optimizes the monitoring solution for in-and-out behavior, and proposes a distributed monitoring system for in-and-out behavior through the combination of machine vision technology and edge computing technology.

[0063] The present application embodiment provides a distributed monitoring system for in-and-out vault behavior, see Figure 1 , Figure 1 A schematic diagram of a system architecture is shown. The system may include a terminal 100 and a server 200. The server 200 may include one or more servers ( Figure 1 In the example, a server is included, and the server 200 can provide the method provided in the embodiment of the present application for one or more terminals.

[0064] Among them, a monitoring application for entering and exiting the safe can be installed on the terminal 100. The above application and web page can provide an interface. The terminal 100 can receive relevant parameters entered by the user on the monitoring interface for entering and exiting the safe, and send the above parameters to the server 200. The server 200 can obtain processing results based on the received parameters and return the processing results to the terminal 100.

[0065] It should be understood that in some optional implementations, the terminal 100 can also complete the action of obtaining the processing result based on the received parameters by itself without the cooperation of the server, and the embodiments of the present application are not limited to this.

[0066] Next describe Figure 1 The product form of the mid-terminal 100;

[0067] The terminal 100 in the embodiment of the present application can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiment of the present application does not impose any limitation on this.

[0068] Figure 2 An optional hardware structure diagram of the terminal 100 is shown.

[0069] refer to Figure 2 As shown, the terminal 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), an earphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190 and other components. Those skilled in the art will appreciate that Figure 2These are merely examples of terminals or multi-function devices and do not constitute limitations on the terminals or multi-function devices, which may include more or fewer components than those shown in the figures, or combinations of certain components, or different components.

[0070] The input unit 130 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the portable multifunctional device. Specifically, the input unit 130 may include a touch screen 131 (optional) and / or other input devices 132. The touch screen 131 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute the command sent by the processor 170; the touch signal at least includes the touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0071] Among them, the input device 132 can receive input data and the like.

[0072] The display unit 140 can be used to display information input by the user or provided to the user, various menus of the terminal 100, interactive interfaces, file displays and / or playback of any multimedia files. In the embodiment of the present application, the display unit 140 can be used to display the monitoring interface of the vault entry and exit behavior, processing results, etc.

[0073] The memory 120 can be used to store instructions and data. The memory 120 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc.; the instruction storage area can store software units such as operating systems, applications, instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; provide the processor 170 with hardware, software and data resources including management of computing and processing equipment, and support control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0074] The processor 170 is the control center of the terminal 100. It uses various interfaces and lines to connect various parts of the entire terminal 100. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it executes various functions of the terminal 100 and processes data, thereby controlling the terminal device as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 170. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 170 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0075] Among them, the memory 120 can be used to store software codes related to the monitoring method of the entry and exit behavior of the safe, the processor 170 can execute the steps of the monitoring method of the entry and exit behavior, and can also schedule other units (such as the above-mentioned input unit 130 and display unit 140) to realize corresponding functions.

[0076] The radio frequency unit 110 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 170 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0077] In this embodiment of the present application, the RF unit 110 can send data to the server 200 and receive processing results sent by the server 200.

[0078] It should be understood that the radio frequency unit 110 is optional and can be replaced by other communication interfaces, such as a network port.

[0079] The terminal 100 also includes a power supply 190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0080] The terminal 100 further includes an external interface 180 , which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal 100 to communicate with other devices, or to connect a charger to charge the terminal 100 .

[0081] Although not shown, the terminal 100 may also include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which are not described in detail here. Some or all of the methods described below may be applied in the following embodiments. Figure 2 In the terminal 100 shown.

[0082] Next describe Figure 1 The product form of the server 200;

[0083] Figure 3 A structural diagram of a server 200 is provided, such as Figure 3 As shown, the server 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate with each other via the bus 201.

[0084] The bus 201 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0085] The processor 202 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0086] The memory 204 may include a volatile memory, such as a random access memory (RAM). The memory 204 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard drive (HDD), or a solid state drive (SSD).

[0087] Among them, the memory 204 can be used to store software codes related to the monitoring method of the in-and-out behavior of the safe, the processor 202 can execute the steps of the monitoring method of the in-and-out behavior of the chip, and can also schedule other units to realize corresponding functions.

[0088] It should be understood that the above-mentioned terminal 100 and server 200 can be centralized or distributed devices, and the processors in the above-mentioned terminal 100 and server 200 (such as processor 170 and processor 202) can be hardware circuits (such as application specific integrated circuit (ASIC), field-programmable gate array (FPGA), general-purpose processor, digital signal processor (DSP), microprocessor or microcontroller, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0089] Figure 4 A specific structural diagram of a monitoring system for entering and exiting a vault provided in an embodiment of the present application, the monitoring system includes a model training platform, an algorithm development module, a normalized operation edge box, a vault monitoring camera, and a cloud platform. Among them, the normalized operation edge box is deployed on the terminal device on the edge side to implement the deployment operation, the terminal device is connected to the vault monitoring camera pre-deployed in the vault door operation area in the vault, and the cloud platform is deployed on the cloud server. The model training platform and the algorithm development module are deployed on a server (including but not limited to a cloud server) to implement the development method.

[0090] The model training platform is used to implement the data collection and processing module, data annotation module, model training module, and model testing and format conversion module.

[0091] The data acquisition and processing module is used to: acquire a safe door image data set and a personnel image data set, wherein the safe door image data set includes safe door images in different states, and the personnel image data set includes personnel images in different states.

[0092] The data labeling module is used to set a safe door status label set and a personnel label set, collect safe door status data based on the safe door status label set to obtain a safe door recognition training set, and obtain a personnel recognition training set based on the personnel image data set of the personnel status label set.

[0093] The model training module is used to: train the machine learning model based on the safe door recognition training set to obtain the safe door recognition model, and train the machine learning model based on the personnel recognition training set to obtain the personnel recognition model.

[0094] The model testing and format conversion module is used to obtain the model test results for the trained vault door recognition model and personnel recognition model, and optimize the vault door recognition model and personnel recognition model based on the model test results until the model test conditions are met. The recognition model is converted into a cross-platform universal onnx format and deployed on the edge box, so that the deployment of the recognition model can be carried out according to the architecture platform of the terminal device on the edge side. The corresponding model conversion and tailoring of the recognition model can be fully utilized. The computing performance of different hardware platforms.

[0095] The algorithm development module includes the entry and exit behavior analysis algorithm development module and the early warning rule judgment algorithm development module.

[0096] The entry and exit behavior analysis algorithm is used to:

[0097] (1) Delineating the safe door area: Delineating the area of ​​each safe door in the surveillance image, and mapping the safe door into a two-dimensional quadrilateral in the surveillance image, that is, the safe door plane quadrilateral.

[0098] (2) Establishing two-dimensional coordinates: Establishing a two-dimensional safe door coordinate system with the bottom vertex of the safe door plane quadrilateral relative to the horizontal line of the surveillance image and the left vertex relative to the vertical line of the surveillance image as coordinate axes to obtain the two-dimensional coordinates of the safe door.

[0099] (3) Person selection: Obtain the rectangular frame position information of the target person output by the person recognition model, and determine whether the target person enters or exits the safe door based on the rectangular frame position information and the coordinates of the vertices of the plane quadrilateral of the safe door.

[0100] Specifically, the rectangular frame position information of the target person is recorded as (x, y, w, h), where x and y are the coordinates of the upper left vertex, and w and h are the width and height of the rectangular frame, respectively.

[0101] Based on the rectangular frame position information and the vertex coordinates of the vault door plane quadrilateral, the method for judging whether the target person triggers the action of entering or exiting the vault door specifically includes:

[0102] 1. Calculate the coordinates of the center point of the rectangular frame of the target person, map the coordinates of the center point to the two-dimensional safe door coordinate system, and then determine whether the center point of the rectangular frame is within the plane quadrilateral of the safe door. If so, it is determined that the target person has triggered the action of entering or exiting the safe door.

[0103] 2. In order to avoid the recognition error caused by the misalignment in the surveillance image or the vault door being an irregular quadrilateral in the surveillance image, the rectangular frame position information of the target person (x, y, w, h) is converted into an edge coordinate set (x, x1+w, x+(w / 2), x+h+(w / 2)), as follows:

[0104] (x,y,w,h)→x,x+w,x+(w / 2),x+h+(w / 2)

[0105] Among them, x in the edge coordinate set is the horizontal coordinate of the left line of the rectangular box, x+w is the horizontal coordinate of the right line of the rectangular box, x+(w / 2) is the vertical coordinate of the upper line of the rectangular box, and x+h+(w / 2) is the vertical coordinate of the lower line of the rectangular box. Each coordinate in the edge coordinate set is converted into the coordinate in the two-dimensional safe door coordinate system to obtain the edge coordinate set.

[0106] Determine whether the horizontal coordinate of the left line of the rectangular box, the horizontal coordinate of the right line of the rectangular box, the vertical coordinate of the upper line of the rectangular box and the vertical coordinate of the lower line of the rectangular box in the edge coordinate set are all within the plane quadrilateral of the safe door. If so, it is considered that the target person has triggered the two-dimensional plane of the safe door, that is, the target person has triggered the action of entering and exiting the safe door.

[0107] The warning rule judgment algorithm is used to judge whether the safe door status information, personnel status information, and action judgment results of each monitoring image in the continuous time image sequence meet the safe entry and exit rules. If not, an early warning is triggered, and the early warning information is generated and sent to the cloud platform.

[0108] The normal operation edge box includes the stream data processing layer, reasoning layer, logic processing layer and warning push layer.

[0109] The stream data processing layer is used to: receive the surveillance video stream of the vault door in real time through the vault surveillance camera channel, decode, extract frames, preprocess data, etc., and convert the video stream into a video image that meets the model input requirements.

[0110] The inference layer is used to load the safe door recognition model and the personnel recognition model at the same time, trigger the safe door model to perform safe door recognition on the monitoring image to obtain the safe door recognition result, and when the safe door recognition result indicates that the open and close state of the safe door has changed, the personnel recognition model is triggered to perform personnel recognition on the monitoring image and output the personnel recognition result. The safe door recognition result and the personnel recognition result are output to the logic processing layer.

[0111] The inference layer uses a time-division multiplexing mechanism to implement multi-model inference operations, dividing continuous time into time slices of fixed length, and alternately running the vault door recognition model and the personnel recognition model in adjacent time slices to achieve the purpose of reasonable use of computing power. Loading two models to the box inference layer at the same time helps shorten the model switching time and reduce the performance loss of model switching.

[0112] The logic processing layer is used to execute the deployed algorithm, perform behavior analysis and early warning rule judgment based on the vault door recognition results and personnel recognition results output by the inference layer, obtain the vault entry and exit rule judgment results, and trigger an early warning when the vault entry and exit rules are not met, generate early warning information and send it to the cloud platform. Among them, the early warning information includes early warning images, vault entry and exit rules, vault door recognition results, personnel recognition results, and violation time. Among them, the early warning image is obtained by drawing a rectangular box of the vault door and the target person on the monitoring image.

[0113] The warning push layer is used to package and push warning information to the cloud platform, so that it can receive, process and store warning information sent by each edge box, and send the alarm information to the management client through SMS, phone, enterprise WeChat and other channels.

[0114] The embodiment of the present application provides a monitoring system for in-and-out vault behavior, which monitors the in-and-out vault door behavior in real time through an edge box deployed in a terminal device on the edge side, and identifies in real time whether the in-and-out vault door behavior complies with the vault in-and-out rules, and issues timely warnings for non-compliant behaviors. The model training platform deployed on the server side is used to train and optimize the recognition model, and the model is sent to each terminal device. The algorithm development module deployed on the server side is used to develop the behavior analysis algorithm and the early warning rule judgment algorithm, and the algorithm is deployed to each terminal device.

[0115] Therefore, the pre-deployed safe door recognition model and personnel recognition model are used to perform safe door recognition and personnel recognition on the surveillance images, thereby improving the real-time performance of safe door recognition and personnel recognition. The distributed deployment of edge-side terminals reduces the computing pressure of the server and avoids the leakage or malicious use of sensitive image information, thereby improving data security.

[0116] Figure 5 A specific implementation flow chart of a method for monitoring in and out of a vault provided in an embodiment of the present application, such as Figure 5 As shown, this method specifically includes:

[0117] S501: Obtain an original data set.

[0118] In this embodiment, the original data set includes a safe door image data set and a personnel image data set, wherein the safe door image data set includes safe door images in different states, and the personnel image data set includes personnel images in different states.

[0119] S502: Process the original data set to obtain a training data set.

[0120] In this embodiment, the process of processing the original data set includes:

[0121] 1. Preprocess the data, including pre-processing operations such as deduplication, standardization and screening.

[0122] 2. Label the preprocessed data to obtain the training data set.

[0123] In this embodiment, the training data set includes a safe door training data set and a personnel training data set.

[0124] The vault door images in the vault door image dataset are annotated to obtain the vault door training dataset. The vault door annotation includes switch state annotation and vault door contour annotation. The vault door contour annotation is the coordinates of each key point of the vault door. The personnel images in the personnel image dataset are annotated to obtain the personnel training dataset. The personnel annotation includes personnel type and personnel position annotation. The personnel position annotation is the vertex set of the circumscribed rectangular box of the personnel image.

[0125] S503: Through the pre-built model training platform, the machine learning model is trained based on the training data set to obtain the safe door recognition model and the personnel recognition model respectively, and the safe door recognition model and the personnel recognition model are deployed to the edge box.

[0126] In this embodiment, the safe door recognition model is used to output the safe door recognition results, which include the safe door open and close status and the safe door key point coordinate set. The personnel recognition model outputs the personnel recognition results, which include the personnel type and personnel location information. The personnel location information includes the vertex coordinate set of the personnel rectangular box, or the target vertex coordinates and length and width values.

[0127] In this embodiment, a vault door recognition model is trained based on a vault door training data set until the training completion condition is met, and various model parameters of the vault door recognition model are obtained, and the vault door recognition model is deployed on an edge box. A personnel recognition model is trained based on a personnel training data set until the training completion condition is met, and various model parameters of the personnel recognition model are obtained, and the personnel recognition model is deployed on an edge box.

[0128] In this embodiment, the edge box is configured in the terminal device at the edge side.

[0129] It should be noted that S501 to 503 are executed by a server, which can connect to multiple bank systems, collect as much raw data as possible, rely on powerful computing power to improve the quality of the raw data, obtain large-volume and high-quality training data, and improve the accuracy of the risk identification model.

[0130] S504, sequentially inputting the surveillance images in the surveillance image sequence into the safe door recognition model in the edge box, obtaining the safe door recognition results of the surveillance images output by the safe door recognition model, and generating a safe door switch state sequence.

[0131] In this embodiment, the surveillance images within the fixed time window are framed in real time to form a surveillance image sequence, that is, a surveillance image sequence. The surveillance image sequence includes surveillance images to be identified arranged in time order, and the surveillance image sequence is updated based on time and identification progress, and surveillance images whose door opening status has not changed are deleted. The surveillance images in the surveillance image sequence are sequentially input into the safe door recognition model in the edge box to obtain the safe door recognition results of the surveillance images output by the safe door recognition model, and generate a safe door switch state sequence. The safe door switch state sequence includes the safe door switch states of the surveillance images arranged in sequence.

[0132] S505. After determining that the switch state of the safe door has changed based on the switch state sequence of the safe door, starting from the target monitoring image, the monitoring images in the monitoring image sequence are input into the personnel recognition model in the edge box to obtain the personnel recognition result output by the personnel recognition model.

[0133] Among them, the target monitoring image is a monitoring image for determining whether the switch state of the safe door has changed. This step configures the trigger logic of the safe door recognition model and the personnel recognition model. After the safe door recognition result indicates that the switch state of the safe door has changed, the safe door recognition result is closed and the personnel recognition model is started. This improves the model detection efficiency, reduces the consumption of model computing resources, and alternately runs the safe door recognition model and the personnel recognition model to achieve the purpose of rationally utilizing computing power.

[0134] S506. Generate a set of coordinates of the vertices of the safe door quadrilateral based on the set of coordinates of the key points of the safe door.

[0135] In this embodiment, the vertex coordinate set of the safe door quadrilateral includes the coordinates of each vertex of the safe door's plane outline. Optionally, the surveillance image is divided into regions, and the safe door is mapped into a two-dimensional quadrilateral in the surveillance image, that is, the safe door's plane quadrilateral outline. Based on the safe door key point coordinate set, a vertex coordinate set of the safe door quadrilateral is generated.

[0136] S507. Establish a two-dimensional safe door coordinate system with a horizontal line passing through the bottom vertex of the safe door plane quadrilateral and a vertical line passing through the left vertex of the safe door plane quadrilateral as coordinate axes, and map the vertex coordinate set of the safe door plane quadrilateral to the two-dimensional safe door coordinate system to obtain a two-dimensional coordinate set of the safe door.

[0137] In this embodiment, the safe door two-dimensional coordinate set includes the coordinates of each vertex of the safe door plane quadrilateral in the two-dimensional safe door coordinate system.

[0138] S508. Obtain the personnel position information of the personnel monitoring image output by the personnel recognition model, obtain the edge coordinate set of the personnel rectangular frame based on the personnel position information, map the edge coordinate set to the two-dimensional safe door coordinate system, and obtain the two-dimensional personnel position information.

[0139] In this embodiment, the person position information is recorded as (x, y, w, h), where x and y are the coordinates of the upper left vertex of the person rectangular box, and w and h are the width and height of the rectangular box respectively. The horizontal side of the person rectangular box is parallel to the horizontal side of the monitoring image.

[0140] Convert the personnel location information (x, y, w, h) into an edge coordinate set (x, x1+w, x+(w / 2), x+h+(w / 2)), as follows:

[0141] (x,y,w,h)→x,x+w,x+(w / 2),x+h+(w / 2)

[0142] Among them, x in the edge coordinate set is the horizontal coordinate of the left line of the rectangular box, x+w is the horizontal coordinate of the right line of the rectangular box, x+(w / 2) is the vertical coordinate of the upper line of the rectangular box, and x+h+(w / 2) is the vertical coordinate of the lower line of the rectangular box. Each coordinate in the edge coordinate set is converted into the coordinate in the two-dimensional safe door coordinate system to obtain a two-dimensional edge coordinate set, that is, the two-dimensional personnel location information.

[0143] S509. Based on the edge coordinate set and the safe door two-dimensional coordinate set, determine whether the target person triggers the safe door entry and exit action, and generate an action judgment result.

[0144] In this embodiment, it is determined whether the horizontal coordinate of the left line of the rectangular box, the horizontal coordinate of the right line of the rectangular box, the vertical coordinate of the upper edge line of the rectangular box and the vertical coordinate of the lower edge line of the rectangular box in the edge coordinate set are all within the plane quadrilateral of the safe door. If so, it is considered that the target person has triggered the two-dimensional plane of the safe door, that is, the target person has triggered the action of entering and exiting the safe door.

[0145] S510: Based on the personnel recognition results of each personnel monitoring image in the continuous time image sequence and the action determination result sequence, determine whether it complies with the pre-configured safe entry and exit rules to obtain the rule determination result.

[0146] In this embodiment, the personnel monitoring image is the monitoring image after the target monitoring image, that is, the monitoring image input into the personnel recognition result. The vault entry and exit rules at least include the specified number of personnel, the specified personnel type, and the specified time. The specified time refers to the maximum interval between the first person's entry and exit time and the last person's entry and exit time.

[0147] In this embodiment, the action determination result sequence includes the action determination results and occurrence times of each surveillance image arranged in sequence.

[0148] In this embodiment, if the type of person who triggers the action of entering and exiting the safe door does not belong to the specified personnel type, it is determined that he does not comply with the safe entry and exit rules. If he does, the person who triggers the action of entering and exiting the safe door will be marked as a compliant person.

[0149] If the type of personnel who triggers the action of entering and exiting the safe door belongs to the specified type of personnel, then it is determined whether the number of compliant personnel meets the specified number of personnel, and whether all compliant personnel trigger the action of entering and exiting the safe door within the specified time.

[0150] Optionally, whether the interval from the first compliance personnel to the last compliance personnel is within the specified time period, if so, it is determined that all compliance personnel trigger the action of entering and exiting the safe door within the specified time period.

[0151] S511. If it does not meet the requirements, an early warning is triggered, and early warning information is generated and sent to the cloud platform.

[0152] In this embodiment, the warning information includes a warning image, safe entry and exit rules, safe door recognition results, personnel recognition results, and violation time, wherein the warning image is obtained by drawing a safe door and a target person rectangular frame on the monitoring image.

[0153] S512. When the preset personnel identification time threshold is reached or the rule judgment result is that the safe entry and exit rules are not met, the personnel identification model is closed and the safe door identification model is opened.

[0154] It can be seen from the above technical solutions that the embodiment of the present application provides a method for monitoring the behavior of entering and exiting the safe, which uses the pre-deployed safe door recognition model and personnel recognition model to perform safe door recognition and personnel recognition on the monitoring image, and makes rule judgments based on the safe door recognition results and personnel recognition results. When there is non-compliance, an early warning is issued to achieve real-time monitoring and real-time alarms, which can significantly reduce the risk and labor costs of manual supervision in this scenario, reduce time loss and solve the lag problem. The distributed deployment of edge-side terminals reduces the computing pressure of the server and avoids the leakage or malicious use of sensitive image information, thereby improving data security.

[0155] Furthermore, this multi-model switching method ensures that only one recognition model is running at any time, reducing the computing load of the equipment and improving operating efficiency. Loading the safe door recognition model and the personnel recognition model into the edge box at the same time is conducive to shortening the model switching time and reducing the performance loss of model switching.

[0156] Furthermore, a two-dimensional model is built for the safe door, and after the three-dimensional safe door contact surface is mapped in two dimensions, the three-dimensional door opening, door entry and door closing actions are detected in two dimensions, and the actions of people entering and exiting the safe door are identified and marked, so as to detect the number and time of people entering and exiting the safe door. Two-dimensional coordinates are established based on the safe door contact surface, and the person target detection frame is converted to the two-dimensional coordinates for calculation, so as to improve the recognition accuracy of the safe door entry and exit actions and reduce the operating load.

[0157] It should be noted that the above is only an optional specific implementation process of a method for monitoring the in-and-out behavior of a safe provided in this embodiment. The present application can also be implemented through other specific implementation methods, for example, the monitoring image is simultaneously input into the safe door recognition model and the personnel recognition model in the edge box to obtain the safe door recognition result output by the safe door recognition model and the personnel recognition result output by the personnel recognition model. For another example, the model training platform can be built on a server or on a terminal device.

[0158] In summary, Figure 6 A flow chart of a method for monitoring in and out of a vault provided in an embodiment of the present application is shown as follows: Figure 6 As shown, the method is applied to a terminal, and the method includes:

[0159] S601. Input the monitoring images in the real-time monitoring image sequence into the safe door recognition model in sequence, and obtain the safe door recognition result output by the safe door recognition model.

[0160] In this embodiment, the safe door recognition result includes the safe door open / closed state and the safe door key point coordinate set.

[0161] In this embodiment, the surveillance images within the fixed time window are framed in real time to form a surveillance image sequence, that is, a surveillance image sequence, which includes surveillance images to be identified arranged in time order, and the surveillance image sequence is updated based on time and identification progress. The safe door switch state sequence includes the safe door switch states of the surveillance images arranged in sequence.

[0162] It should be noted that the safe door opening and closing status in the safe door identification result specifically includes the safe door closing status or opening degree.

[0163] S602. If a change in the switch state of the safe door is detected, starting from the target monitoring image, the monitoring images in the real-time monitoring image sequence are sequentially input into the personnel recognition model to obtain the personnel recognition result output by the personnel recognition model.

[0164] In this embodiment, the personnel identification result includes the personnel type and personnel location information, and the target monitoring image is a monitoring image of changes in the door opening and closing states of the safe door.

[0165] In this embodiment, the safe door recognition model and the personnel recognition model are both obtained by the server training a machine learning model based on a training data set.

[0166] S603. Based on the target monitoring image, construct a two-dimensional coordinate system in the safe door plane as the two-dimensional safe door coordinate system.

[0167] S604: Map the safe door key point coordinate set of the target monitoring image to a two-dimensional safe door coordinate system to obtain a two-dimensional safe door coordinate set.

[0168] S605. For each personnel monitoring image, map the personnel position information of the personnel monitoring image to a two-dimensional safe door coordinate system to obtain two-dimensional personnel position information.

[0169] In this embodiment, the personnel monitoring image is a monitoring image input into the personnel recognition model.

[0170] S606. For each personnel monitoring image, generate an action judgment result based on the two-dimensional coordinate set of the safe door and the two-dimensional personnel position information.

[0171] In this embodiment, the action judgment result indicates whether the target person in the personnel monitoring image triggers the action of entering or exiting the safe door.

[0172] S607: Based on the personnel recognition results and action determination result sequences of each personnel monitoring image, determine whether it complies with the pre-configured safe entry and exit rules.

[0173] In this embodiment, the action determination result sequence includes the action determination results of the personnel monitoring images arranged in chronological order.

[0174] In this embodiment, the vault entry and exit rules at least include the specified number of personnel, the specified personnel type, and the specified time, wherein the specified time refers to the maximum interval between the first person's entry and exit time and the last person's entry and exit time.

[0175] In an optional embodiment, if the type of person who triggers the action of entering and exiting the safe door does not belong to the specified personnel type, it is determined that it does not comply with the safe door entry and exit rules. If it does, the person who triggers the action of entering and exiting the safe door is marked as a compliant person. If the type of person who triggers the action of entering and exiting the safe door belongs to the specified personnel type, it is determined whether the number of compliant persons meets the specified number of persons, and all compliant persons trigger the action of entering and exiting the safe door within the specified time. Optionally, whether the interval time from the first compliant person to the last compliant person is within the specified time. If so, it is determined that all compliant persons trigger the action of entering and exiting the safe door within the specified time.

[0176] S608. If the vault entry and exit rules are not met, a warning message is issued.

[0177] It can be seen from the above technical scheme that an embodiment of the present application provides a method for monitoring safe entry and exit behaviors, which utilizes a pre-deployed safe door recognition model and a personnel recognition model to perform safe door recognition and personnel recognition on monitoring images, and constructs a two-dimensional coordinate system in the safe door plane based on the target monitoring image as a two-dimensional safe door coordinate system, and generates an action judgment result based on the safe door two-dimensional coordinate set and the two-dimensional personnel position information, and judges whether the pre-configured safe entry and exit rules are met based on the personnel recognition results of each of the personnel monitoring images and the sequence of action judgment results, and issues an early warning when it is not compliant. This scheme improves the real-time monitoring of safe entry and exit behaviors and improves the accuracy of action judgment through two-dimensional mapping.

[0178] It should be noted that the monitoring method and related device for entering and exiting a safe provided by the present invention can be used in the field of artificial intelligence or finance. The above is only an example and does not limit the application field of the monitoring method and related device for entering and exiting a safe provided by the present invention.

[0179] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0180] A method for monitoring the entry and exit of a safe provided in an embodiment of the present application is introduced above. A device for executing the above method for monitoring the entry and exit of a safe will be introduced below. Figure 7 A schematic diagram of a monitoring device for entering and exiting a safe provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the device comprises:

[0181] The safe door recognition unit 701 is used to sequentially input the monitoring images in the real-time monitoring image sequence into the safe door recognition model to obtain the safe door recognition result output by the safe door recognition model; the safe door recognition result includes the safe door opening and closing state and the safe door key point coordinate set;

[0182] The personnel identification unit 702 is used to input the monitoring images in the real-time monitoring image sequence into the personnel identification model in sequence, starting from the target monitoring image, to obtain the personnel identification result output by the personnel identification model, wherein the personnel identification result includes the personnel type and personnel location information, and the target monitoring image is a monitoring image in which the door opening and closing state of the safe door changes; wherein the safe door identification model and the personnel identification model are both obtained by training a machine learning model based on a training data set by the server;

[0183] A two-dimensional coordinate system construction unit 703 is used to construct a two-dimensional coordinate system in the safe door plane based on the target monitoring image as a two-dimensional safe door coordinate system;

[0184] A two-dimensional coordinate mapping unit 704 is used to map the safe door key point coordinate set of the target monitoring image to the two-dimensional safe door coordinate system to obtain a two-dimensional safe door coordinate set; for each personnel monitoring image, the personnel position information of the personnel monitoring image is mapped to the two-dimensional safe door coordinate system to obtain two-dimensional personnel position information, and the personnel monitoring image is the monitoring image input into the personnel recognition model;

[0185] The action judgment unit 705 is used to generate an action judgment result for each of the personnel monitoring images based on the safe door two-dimensional coordinate set and the two-dimensional personnel position information, wherein the action judgment result indicates whether the target person in the personnel monitoring image triggers an action of entering or exiting the safe door;

[0186] A rule judgment unit 706, configured to judge whether a pre-configured safe entry and exit rule is met based on the personnel recognition result and the action judgment result sequence of each of the personnel monitoring images, wherein the action judgment result sequence includes the action judgment results of the personnel monitoring images arranged in chronological order;

[0187] The warning unit 707 is used to issue a warning message if the vault entry and exit rules are not met.

[0188] Optionally, the monitoring device for entering and exiting the safe also includes a switch status judgment unit, which is used to update the safe door switch status sequence after obtaining each safe door identification result output by the safe door identification model, and the safe door switch status sequence includes the safe door switch status of each monitoring image arranged in chronological order; based on the safe door switch status sequence, determine whether the safe door switch status has changed.

[0189] Optionally, the two-dimensional coordinate system construction unit is used to construct a two-dimensional coordinate system in the safe door plane based on the target monitoring image, and when used as a two-dimensional safe door coordinate system, it is specifically used for:

[0190] Based on the safe door key point coordinate set, generate a safe door vertex coordinate set;

[0191] A two-dimensional coordinate system in the vault door plane is established with a horizontal line passing through the lowest vertex in the vault door vertex coordinate set and a vertical line passing through the leftmost vertex in the vault door vertex coordinate set as coordinate axes as the two-dimensional vault door coordinate system;

[0192] Optionally, the two-dimensional coordinate mapping unit is used to map the safe door key point coordinate set of the target monitoring image to the two-dimensional safe door coordinate system, and when the two-dimensional coordinate set of the safe door is obtained, it is specifically used to:

[0193] The coordinates of each vertex in the vault door vertex coordinate set are mapped to the two-dimensional vault door coordinate system to obtain the two-dimensional vault door coordinate set.

[0194] Optionally, the two-dimensional coordinate mapping unit is used to map the personnel position information of the personnel monitoring image to the two-dimensional safe door coordinate system, and the two-dimensional personnel position information is obtained specifically for:

[0195] Acquire a side coordinate set of a person rectangular frame based on the person position information, wherein the side coordinate set includes a longitudinal coordinate of each horizontal side and a transverse coordinate of each vertical side of the person rectangular frame;

[0196] The edge coordinate set of the personnel rectangular frame is mapped to the two-dimensional safe door coordinate system to obtain the two-dimensional personnel position information.

[0197] Optionally, the action judgment unit is used to generate an action judgment result based on the two-dimensional coordinate set of the safe door and the two-dimensional personnel position information, specifically for:

[0198] Determine whether the rectangle formed by each coordinate in the two-dimensional personnel position information is within the outline formed by each vertex in the two-dimensional coordinate set of the safe door;

[0199] If so, it is determined that the person triggers the action of entering or exiting the safe door.

[0200] Optionally, the monitoring device for the in-and-out behavior further includes a model switching unit, which is used to:

[0201] Loading the safe door recognition model and the personnel recognition model, and setting the initial states of the safe door recognition model and the personnel recognition model to closed;

[0202] If a change in the switch state of the safe door is detected, the safe door recognition model is turned off and the personnel recognition model is turned on;

[0203] When the preset time threshold is reached or the rule determination result is non-compliance, the personnel recognition model is closed and the safe door recognition model is opened.

[0204] The present application also provides an electronic device in an embodiment. Figure 8 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic device in the embodiments of the present application may include but is not limited to fixed terminals such as mobile phones, notebook computers, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 8 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0205] like Figure 8 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 to a random access memory (RAM) 803. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 803. The processing device 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0206] Typically, the following devices may be connected to the I / O interface 805: input devices 806 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 808 including, for example, a memory card, a hard disk, etc.; and communication devices 809. The communication device 809 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0207] Also provided in an embodiment of the present application is a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any method for monitoring the entry and exit of a safe provided in an embodiment of the present application.

[0208] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When one or more computer programs are executed by an electronic device, the electronic device can implement any method for monitoring entry and exit of the safe provided in an embodiment of the present application.

[0209] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0210] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0211] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0212] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A method for monitoring the behavior of entering and leaving a safe, characterized in that: Applied to terminals, including: The monitoring images in the real-time monitoring image sequence are sequentially input into the safe door recognition model to obtain the safe door recognition result output by the safe door recognition model; the safe door recognition result includes the safe door opening and closing state and the safe door key point coordinate set; If a change in the switch state of the safe door is detected, starting from the target monitoring image, the monitoring images in the real-time monitoring image sequence are sequentially input into the personnel recognition model to obtain the personnel recognition result output by the personnel recognition model, the personnel recognition result includes the personnel type and personnel location information, and the target monitoring image is a monitoring image of the change in the switch state of the safe door; wherein the safe door recognition model and the personnel recognition model are both obtained by the server training a machine learning model based on a training data set; Based on the target monitoring image, construct a two-dimensional coordinate system in the safe door plane as a two-dimensional safe door coordinate system; Mapping the safe door key point coordinate set of the target monitoring image to the two-dimensional safe door coordinate system to obtain a two-dimensional safe door coordinate set; For each personnel monitoring image, mapping the personnel position information of the personnel monitoring image to the two-dimensional safe door coordinate system to obtain the two-dimensional personnel position information, wherein the personnel monitoring image is the monitoring image input into the personnel recognition model; For each of the personnel monitoring images, based on the two-dimensional coordinate set of the safe door and the two-dimensional personnel position information, an action judgment result is generated, wherein the action judgment result indicates whether the target person in the personnel monitoring image triggers an action of entering or exiting the safe door; Based on the personnel recognition results and action determination result sequences of each of the personnel monitoring images, judging whether the pre-configured vault entry and exit rules are met, the action determination result sequence comprising the action determination results of the personnel monitoring images arranged in chronological order; If the vault entry and exit rules are not met, an early warning message will be issued.

2. The method for monitoring the in and out behavior of a safe according to claim 1, characterized in that: The method for monitoring the in-and-out behavior of the vault also includes: After obtaining the identification results of each safe door output by the safe door identification model, updating the safe door switch state sequence, wherein the safe door switch state sequence includes the safe door switch states of each monitoring image arranged in chronological order; Determine whether the safe door switch state changes based on the safe door switch state sequence.

3. The method for monitoring the in and out behavior of a safe according to claim 1, characterized in that: The method of constructing a two-dimensional coordinate system in the safe door plane based on the target monitoring image as a two-dimensional safe door coordinate system includes: Based on the safe door key point coordinate set, generate a safe door vertex coordinate set; A two-dimensional coordinate system in the vault door plane is established with a horizontal line passing through the lowest vertex in the vault door vertex coordinate set and a vertical line passing through the leftmost vertex in the vault door vertex coordinate set as coordinate axes as the two-dimensional vault door coordinate system; The key point coordinate set of the safe door of the target monitoring image is mapped to the two-dimensional safe door coordinate system to obtain the two-dimensional coordinate set of the safe door, including: The coordinates of each vertex in the vault door vertex coordinate set are mapped to the two-dimensional vault door coordinate system to obtain the two-dimensional vault door coordinate set.

4. The method for monitoring the in-and-out behavior of a safe according to claim 3, characterized in that: Mapping the personnel position information of the personnel monitoring image to the two-dimensional safe door coordinate system to obtain the two-dimensional personnel position information includes: Acquire a side coordinate set of a person rectangular frame based on the person position information, wherein the side coordinate set includes a longitudinal coordinate of each horizontal side and a transverse coordinate of each vertical side of the person rectangular frame; The edge coordinate set of the personnel rectangular frame is mapped to the two-dimensional safe door coordinate system to obtain the two-dimensional personnel position information.

5. The method for monitoring the in and out behavior of a safe according to claim 4, characterized in that: The generating of the action judgment result based on the two-dimensional coordinate set of the safe door and the two-dimensional personnel position information includes: Determine whether the rectangle formed by each coordinate in the two-dimensional personnel position information is within the outline formed by each vertex in the two-dimensional coordinate set of the safe door; If so, it is determined that the person triggers the action of entering or exiting the safe door.

6. The method for monitoring the in and out behavior of a safe according to claim 1, characterized in that: The method for monitoring the in-and-out behavior of the vault further includes: Loading the safe door recognition model and the personnel recognition model, and setting the initial states of the safe door recognition model and the personnel recognition model to closed; If a change in the switch state of the safe door is detected, the safe door recognition model is turned off and the personnel recognition model is turned on; When the preset time threshold is reached or the rule determination result is non-compliance, the personnel recognition model is closed and the safe door recognition model is opened.

7. A monitoring device for entering and leaving a safe, characterized in that: include: A safe door recognition unit is used to sequentially input the monitoring images in the real-time monitoring image sequence into the safe door recognition model to obtain the safe door recognition result output by the safe door recognition model; the safe door recognition result includes the safe door opening and closing state and the safe door key point coordinate set; A personnel identification unit is used to input the monitoring images in the real-time monitoring image sequence into the personnel identification model in sequence, starting from the target monitoring image, to obtain the personnel identification result output by the personnel identification model, wherein the personnel identification result includes the personnel type and personnel location information, and the target monitoring image is a monitoring image in which the door opening and closing state of the safe door changes; wherein the safe door identification model and the personnel identification model are both obtained by training a machine learning model based on a training data set by the server; A two-dimensional coordinate system construction unit, used to construct a two-dimensional coordinate system in the safe door plane based on the target monitoring image as a two-dimensional safe door coordinate system; A two-dimensional coordinate mapping unit is used to map the safe door key point coordinate set of the target monitoring image to the two-dimensional safe door coordinate system to obtain the safe door two-dimensional coordinate set; for each personnel monitoring image, the personnel position information of the personnel monitoring image is mapped to the two-dimensional safe door coordinate system to obtain the two-dimensional personnel position information, and the personnel monitoring image is the monitoring image input into the personnel recognition model; An action judgment unit, for generating an action judgment result for each of the personnel monitoring images based on the two-dimensional coordinate set of the safe door and the two-dimensional personnel position information, wherein the action judgment result indicates whether the target person in the personnel monitoring image triggers an action of entering or exiting the safe door; A rule judgment unit, used to judge whether the pre-configured vault entry and exit rules are met based on the personnel recognition results and action judgment result sequences of each of the personnel monitoring images, wherein the action judgment result sequences include the action judgment results of the personnel monitoring images arranged in chronological order; The early warning unit is used to issue an early warning message if the vault entry and exit rules are not met.

8. A computer program product, characterized in that It comprises computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the method for monitoring the entry and exit behavior of a safe as described in any one of claims 1 to 6.

9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the method for monitoring the entry and exit behavior of the safe as described in any one of claims 1 to 6.

10. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the method for monitoring safe entry and exit behaviors as described in any one of claims 1 to 6.

Images