Vault handover monitoring method and related device

Abstract

The invention discloses a vault handover monitoring method and a related device, which can be applied to the field of artificial intelligence or finance. The vault handover monitoring area is provided with a plurality of camera devices for collecting different visual angles so as to collect video streams of different visual angles in the vault handover monitoring area, and multi-angle monitoring is achieved. Besides, a mapping relation between a two-dimensional image collected by the camera device and a three-dimensional space is determined, based on the mapping relation, video streams collected by the multiple camera devices are projected to the same plane, image processing operation is performed on the video streams projected to the same plane, and an overall monitoring video stream is obtained. The complete video stream of the area needing to be monitored can be completely viewed in the same monitoring video stream, so that the personnel detection result and the article detection result corresponding to the whole monitoring video stream can be determined by using the detection model, and the vault handover monitoring result is determined according to the personnel detection result and the article detection result. The problem of manual monitoring omission is avoided, and the monitoring accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of financial technology, and in particular to a vault handover monitoring method and related devices. Background Art

[0002] The bank vault is a very important security location, which is distributed all over the country and involves many high-standard compliance requirements, regulatory requirements and internal control requirements. The bank has a set of regulations for the types of people and processes entering the vault, and the internal and external personnel of the vault also need to hand over according to relevant rules and regulations.

[0003] At present, when monitoring the handover of the vault, it is generally checked by manually observing the monitoring screen. This method is prone to omissions, resulting in low monitoring accuracy. Summary of the invention

[0004] In view of the above problems, the present application provides a vault handover monitoring method and related devices to achieve the purpose of improving the accuracy of vault handover monitoring. The specific scheme is as follows:

[0005] The first aspect of the present application provides a vault handover monitoring method, comprising:

[0006] Acquire video streams collected by multiple camera devices in the safe handover monitoring area, wherein the safe handover monitoring area is configured with multiple camera devices collecting different viewing angles to collect video streams from different viewing angles of the safe handover monitoring area;

[0007] Determine a mapping relationship between the two-dimensional image captured by the camera device and the three-dimensional space;

[0008] Based on the mapping relationship, the video streams collected by the multi-channel camera devices are projected onto the same plane, and image processing operations are performed on the video streams projected onto the same plane to obtain an overall monitoring video stream;

[0009] A pre-configured detection model is obtained, wherein the training samples used by the detection model during training are: sample data and probability distribution corresponding to the sample data; the probability distribution includes: a first probability distribution and a second probability distribution, wherein the first probability distribution is: a probability distribution corresponding to a target position obtained by inputting the sample data into a preset large model; the second probability distribution is: a probability distribution corresponding to a category prediction result obtained by inputting the sample data into the preset large model; during training of the detection model, the preset parameter optimization method is used to optimize the configuration of model parameters;

[0010] Determine the personnel detection result and the object detection result corresponding to the overall monitoring video stream by using the detection model; the personnel detection result includes the target position and the type of the personnel; the object detection result includes the target position and the type of the object;

[0011] The vault handover monitoring result is determined based on the personnel detection result and the item detection result.

[0012] In a possible implementation, determining a mapping relationship between a two-dimensional image captured by the camera device and a three-dimensional space includes:

[0013] Using the multi-channel camera device to capture images of reference plates at different positions and in different postures, and obtaining the reference plate position coordinates when the reference plates are capturing images;

[0014] Extracting pixel coordinates of corner points of the image;

[0015] Calculating mapping parameters according to the corner point pixel coordinates and the reference plate position coordinates using the camera parameters of the multi-channel camera device;

[0016] A mapping relationship between the two-dimensional picture captured by the camera device and the three-dimensional space corresponding to the mapping parameters is determined.

[0017] In a possible implementation, image processing operations are performed on the video streams projected onto the same plane to obtain an overall monitoring video stream, including:

[0018] Perform image stitching and fusion operations on the video streams projected onto the same plane to obtain the overall monitoring video stream.

[0019] In a possible implementation, the process of optimizing the configuration of model parameters using the preset parameter optimization method includes:

[0020] Determining a weighted objective function and determining a hyperparameter space of hyperparameters;

[0021] Select the Gaussian process surrogate model;

[0022] Determine hyperparameter configuration information using the Gaussian process proxy model and iteration strategy;

[0023] According to the hyperparameter configuration information, the hyperparameter is optimized in the hyperparameter space, and after the optimization operation is completed, the function value of the weighted objective function is calculated;

[0024] If the function value of the weighted objective function does not converge to within a preset threshold range, updating the Gaussian process proxy model based on the function value of the weighted objective function, and returning to the step of determining the hyperparameter configuration information using the Gaussian process proxy model and the iteration strategy;

[0025] If the function value of the weighted objective function converges to within a preset threshold range, the obtained hyperparameters are used as model parameters of the detection model.

[0026] In a possible implementation, using the detection model to determine a person detection result and an object detection result corresponding to the overall surveillance video stream includes:

[0027] Get the current bit rate of multiple camera devices;

[0028] Get current network data;

[0029] Obtain the vault handover monitoring results of the previous inspection cycle;

[0030] Determine a video stream frame extraction rule according to the bit rate of the multi-channel camera device, the current network data, and the vault handover monitoring result detected in the previous detection cycle;

[0031] According to the video stream frame extraction rule, a target video frame is extracted from the overall monitoring video stream, and the detection model is used to determine the person detection result and the object detection result corresponding to the target video frame.

[0032] In a possible implementation, determining the vault handover monitoring result according to the personnel detection result and the item detection result includes:

[0033] Determine whether the object detection result corresponding to the target video frame is a preset object;

[0034] If yes, calculate the total number of different types of people according to the person detection result corresponding to the target video frame;

[0035] Calculate the average of the total number of different types of personnel;

[0036] If the average total number of different types of personnel is greater than the preset threshold, the vault handover monitoring result is determined to be the first indicator of compliance;

[0037] If the average value of the total number of at least one type of personnel is not greater than a preset threshold, the safe handover monitoring result is determined to be a second indicator representing a violation.

[0038] A second aspect of the present application provides a safe handover monitoring device, comprising:

[0039] A video stream acquisition module is used to acquire video streams collected by multiple camera devices in the safe handover monitoring area, wherein the safe handover monitoring area is configured with multiple camera devices that collect different viewing angles to collect video streams from different viewing angles of the safe handover monitoring area;

[0040] A relationship determination module, used to determine the mapping relationship between the two-dimensional picture captured by the camera device and the three-dimensional space;

[0041] An integration module, used for projecting the video streams collected by the multi-channel camera devices onto the same plane based on the mapping relationship, performing image processing operations on the video streams projected onto the same plane, and obtaining an overall monitoring video stream;

[0042] A model acquisition module is used to acquire a pre-configured detection model. The training samples used in the training of the detection model are: sample data and probability distribution corresponding to the sample data; the probability distribution includes: a first probability distribution and a second probability distribution. The first probability distribution is: the probability distribution corresponding to the target position obtained by inputting the sample data into a preset large model; the second probability distribution is: the probability distribution corresponding to the category prediction result obtained by inputting the sample data into the preset large model; when the detection model is trained, the preset parameter optimization method is used to optimize the configuration of model parameters;

[0043] A detection module, used to determine, by using the detection model, a person detection result and an object detection result corresponding to the overall monitoring video stream; the person detection result includes a target position and a type of person; the object detection result includes a target position and a type of object;

[0044] The monitoring determination module is used to determine the vault handover monitoring result based on the personnel detection result and the item detection result.

[0045] A third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the above-mentioned safe handover monitoring method.

[0046] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0047] The memory is used to store computer programs;

[0048] The processor is used to execute the computer program so that the electronic device can implement the above-mentioned safe handover monitoring method.

[0049] A fifth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the above-mentioned safe handover monitoring method.

[0050] By means of the above technical scheme, the present application provides a vault handover monitoring method and related devices. In the present application, the vault handover monitoring area is configured with multiple camera devices that collect different viewing angles to collect video streams of different viewing angles of the vault handover monitoring area, so as to realize multi-angle monitoring, and can avoid the problem of limited monitoring range caused by object occlusion. In addition, in the present application, the mapping relationship between the two-dimensional picture collected by the camera device and the three-dimensional space is determined, and based on the mapping relationship, the video streams collected by the multiple camera devices are projected onto the same plane, and the video streams projected onto the same plane are subjected to image processing operations to obtain an overall monitoring video stream, so that the complete video stream of the required monitoring area can be fully viewed in the same monitoring video stream, so that the detection model can be used to determine the personnel detection results and the object detection results corresponding to the overall monitoring video stream, and the vault handover monitoring results can be determined according to the personnel detection results and the object detection results. In the embodiment of the present invention, since the video stream is used in real time for monitoring operations, the problem of omissions in manual monitoring can be avoided, and the monitoring accuracy can be improved. In addition, in the present application, the training samples used in the training of the detection model are: sample data and the probability distribution corresponding to the sample data; the probability distribution includes: a first probability distribution and a second probability distribution, the first probability distribution is: the probability distribution corresponding to the target position obtained by inputting the sample data into the preset large model; the second probability distribution is: the probability distribution corresponding to the category prediction result obtained by inputting the sample data into the preset large model. Since the training samples use the probability distribution corresponding to the output result of the preset large model, a small-volume model can be used to realize personnel detection and object detection, reducing the resource occupation of the model. In addition, in the present invention, when the detection model is trained, the preset parameter optimization method is used to optimize the configuration of the model parameters, thereby ensuring the accuracy of the parameters in the model and improving the accuracy of the model detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0052] Figure 1 A schematic diagram of a system architecture provided for this application;

[0053] Figure 2 A schematic diagram of a terminal structure provided for this application;

[0054] Figure 3 A schematic diagram of the structure of an edge computing device provided for this application;

[0055] Figure 4A flow chart of a vault handover monitoring method provided for this application;

[0056] Figure 5 A flowchart of a method for determining a mapping relationship provided in this application;

[0057] Figure 6 A coordinate transformation schematic diagram provided for this application;

[0058] Figure 7 A schematic diagram of image integration provided for this application;

[0059] Figure 8 A schematic diagram of a scenario of a vault handover monitoring method provided in this application;

[0060] Fig. 9 A schematic diagram of the structure of a vault handover monitoring device provided in this application;

[0061] Fig.10 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0062] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0063] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0064] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0065] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0066] In order to enable those skilled in the art to understand the present invention, the professional terms in the present invention are now explained.

[0067] Machine vision: refers to the technology that uses computer vision technology to extract useful information from visual perception information such as images and videos and further process it; currently, the application of machine vision in banking vault scenarios focuses on facial identity recognition, person feature recognition, object feature recognition and person trajectory recognition.

[0068] Edge computing: refers to the local deployment at the edge of the network, that is, the computing power user, and all related processing is completed at the edge; only a small amount of key information is transmitted to the server, or no information is transmitted to the server. This method can reduce the requirements of high concurrency and high bandwidth, and achieve faster processing.

[0069] The bank vault is a very important security location, which is distributed all over the country and involves many high-standard compliance requirements, regulatory requirements and internal control requirements. The bank has a set of regulations for the types of people and processes entering the vault, and the internal and external personnel of the vault also need to hand over according to relevant rules and regulations.

[0070] At present, when monitoring the handover of vaults, it is generally checked by manual observation of the monitoring screen, specifically by manually checking through the monitoring screen or replaying the monitoring video afterwards. The staff judges whether there are non-standard and irregular behaviors between the two parties during the handover by subjective judgment with the naked eye and reminds the relevant personnel. This method is easy to miss and has a small coverage. Therefore, the sampling inspection method is adopted in actual operation. It is more difficult to find illegal behaviors, resulting in low monitoring accuracy. In addition, manual spot checks also have the problems of large workload and high labor costs.

[0071] In view of the shortcomings of manual spot checks of monitoring images, such as high cost, heavy workload, easy omissions, small coverage and inability to monitor in real time, the present invention uses machine vision technology and edge computing technology to establish a low-cost and easy-to-deploy early warning system for real-time analysis of monitoring images. By performing real-time analysis on the handover behavior of personnel in the handover area of ​​the monitoring image, it is determined whether the system algorithm logic is met, and then whether there is any violation, thereby realizing real-time early warning of violations. The present invention can reduce the manual workload and inspection lag problems, and can cover all vault environments and all time periods, reducing omissions in manual inspections and errors in subjective assumptions.

[0072] In specific implementation, the safe handover monitoring area is configured with multiple cameras that capture different perspectives to capture video streams from different perspectives in the safe handover monitoring area, thereby achieving multi-angle monitoring and avoiding the problem of limited monitoring range caused by obstruction of objects.

[0073] In addition, in the present application, the mapping relationship between the two-dimensional image captured by the camera device and the three-dimensional space is determined, and based on the mapping relationship, the video streams captured by the multi-channel camera device are projected onto the same plane, and image processing operations are performed on the video streams projected onto the same plane to obtain an overall monitoring video stream, so that the complete video stream of the required monitoring area can be fully viewed in the same monitoring video stream, so that the detection model can be used to determine the personnel detection results and the object detection results corresponding to the overall monitoring video stream, and the vault handover monitoring results are determined based on the personnel detection results and the object detection results. In the embodiment of the present invention, since the video stream is used in real time for monitoring operations, the problem of omissions in manual monitoring can be avoided, and the monitoring accuracy can be improved.

[0074] In addition, in the present application, the training samples used in the training of the detection model are: sample data and the probability distribution corresponding to the sample data; the probability distribution includes: a first probability distribution and a second probability distribution, the first probability distribution is: the probability distribution corresponding to the target position obtained by inputting the sample data into the preset large model; the second probability distribution is: the probability distribution corresponding to the category prediction result obtained by inputting the sample data into the preset large model. Since the training samples use the probability distribution corresponding to the output result of the preset large model, a small-volume model can be used to realize personnel detection and object detection, reducing the resource occupation of the model. In addition, in the present invention, when the detection model is trained, the preset parameter optimization method is used to optimize the configuration of the model parameters, thereby ensuring the accuracy of the parameters in the model and improving the accuracy of the model detection.

[0075] Through the above steps, the present invention can significantly reduce the risk of risk control and labor costs of manual supervision in this scenario, and achieve real-time warning to reduce time loss, quickly discover violations to avoid increased risks; avoid the conventional method that can only detect and discover violations in the vault after a period of time and issue timely reminders.

[0076] In addition, most existing technical solutions adopt the deployment method of central edge computing devices to send events to edge computing devices for unified processing. The present invention uses edge computing technology to be deployed in the vault, directly connects the camera to perform machine vision recognition and then performs secondary logic algorithm processing, and finally realizes edge intelligent monitoring, reducing the task processing volume of the central edge computing device.

[0077] Based on the above content, an embodiment of the present invention provides a vault handover monitoring method and related devices. It should be noted that the vault handover monitoring method and related devices provided by the present invention can be used in the field of artificial intelligence or the financial field. The above is only an example and does not limit the application field of the vault handover monitoring method and related devices provided by the present invention.

[0078] See also Figure 1 , Figure 1 A schematic diagram of a system architecture is shown. The system may include a terminal 100 and an edge computing device 200. The edge computing device 200 may include one or more edge computing devices ( Figure 1 In the example of an edge computing device, the edge computing device 200 can provide the method provided in the embodiment of the present application for one or more terminals.

[0079] Among them, a data acquisition application can be installed on the terminal 100. The above application and web page can provide an interface. The terminal 100 can collect video streams and send the video streams to the edge computing device 200. The edge computing device 200 can obtain processing results based on the received video streams.

[0080] Next describe Figure 1 The product form of the mid-terminal 100;

[0081] The terminal 100 in the embodiment of the present application can be a camera device, a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiment of the present application does not impose any limitation on this.

[0082] Figure 2 An optional hardware structure diagram of the terminal 100 is shown.

[0083] refer to Figure 2 As shown, the terminal 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), an earphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190 and other components. Those skilled in the art will appreciate that Figure 2These are merely examples of terminals or multi-function devices and do not constitute limitations on the terminals or multi-function devices, which may include more or fewer components than those shown in the figures, or combinations of certain components, or different components.

[0084] The input unit 130 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the portable multifunctional device. Specifically, the input unit 130 may include a touch screen 131 (optional) and / or other input devices 132. The touch screen 131 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute the command sent by the processor 170; the touch signal at least includes the touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0085] Among them, the input device 132 can receive input data and the like.

[0086] The display unit 140 may be used to display information input by the user or provided to the user, various menus of the terminal 100, interactive interfaces, file displays, and / or playback of any multimedia file. In an embodiment of the present application, the display unit 140 may be used to display an interface for video stream acquisition, processing results, and the like.

[0087] The memory 120 can be used to store instructions and data. The memory 120 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc.; the instruction storage area can store software units such as operating systems, applications, instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; provide the processor 170 with hardware, software and data resources including management of computing and processing equipment, and support control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0088] The processor 170 is the control center of the terminal 100. It uses various interfaces and lines to connect various parts of the entire terminal 100. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it executes various functions of the terminal 100 and processes data, thereby controlling the terminal device as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 170. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 170 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0089] Among them, the memory 120 can be used to store software codes related to the video stream acquisition method, the processor 170 can execute the steps of the video stream acquisition method, and can also schedule other units (such as the above-mentioned input unit 130 and display unit 140) to achieve corresponding functions.

[0090] The radio frequency unit 110 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 170 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0091] In this embodiment of the present application, the radio frequency unit 110 can send data to the edge computing device 200.

[0092] It should be understood that the radio frequency unit 110 is optional and can be replaced by other communication interfaces, such as a network port.

[0093] The terminal 100 also includes a power supply 190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0094] The terminal 100 further includes an external interface 180 , which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal 100 to communicate with other devices, or to connect a charger to charge the terminal 100 .

[0095] Although not shown, the terminal 100 may also include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which are not described in detail here. Some or all of the methods described below may be applied in the following embodiments. Figure 2 In the terminal 100 shown.

[0096] Next describe Figure 1 The product form of the edge computing device 200;

[0097] Figure 3 A schematic diagram of the structure of an edge computing device 200 is provided, such as Figure 3 As shown, the edge computing device 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate with each other through the bus 201.

[0098] The bus 201 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0099] The processor 202 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0100] The memory 204 may include a volatile memory, such as a random access memory (RAM). The memory 204 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard drive (HDD), or a solid state drive (SSD).

[0101] The memory 204 may be used to store software codes related to the safe handover monitoring method, the processor 202 may execute the steps of the safe handover monitoring method of the chip, and may also schedule other units to implement corresponding functions.

[0102] It should be understood that the above-mentioned terminal 100 and edge computing device 200 can be centralized or distributed devices, and the processors in the above-mentioned terminal 100 and edge computing device 200 (such as processor 170 and processor 202) can be hardware circuits (such as application specific integrated circuit (ASIC), field-programmable gate array (FPGA), general-purpose processor, digital signal processor (DSP), microprocessor or microcontroller, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0103] In order to solve the above problems, the embodiment of the present application provides a safe handover monitoring method. The safe handover monitoring method of the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0104] Reference Figure 4 , Figure 4 A flow chart of a vault handover monitoring method provided in an embodiment of the present application is shown as follows: Figure 4 As shown, a vault handover monitoring method provided by an embodiment of the present application may include:

[0105] S11, obtaining video streams collected by multiple cameras in the safe handover monitoring area.

[0106] In actual scenarios, since the safe handover monitoring area is large, it is impossible to fully monitor the area with only one camera device, and the area may be blocked by items, resulting in an incomplete monitoring range. In order to avoid this problem, the safe handover monitoring area is equipped with multiple cameras that collect different perspectives to collect video streams from different perspectives of the safe handover monitoring area. Subsequently, the video streams collected by all cameras are fused to obtain a complete monitoring video stream of the safe handover monitoring area.

[0107] S12: Determine a mapping relationship between the two-dimensional image captured by the camera device and the three-dimensional space.

[0108] Specifically, in order to achieve the fusion of video streams captured by multiple cameras, the video streams captured by the cameras need to be converted from two-dimensional images to three-dimensional space. During the conversion, the mapping relationship between the two-dimensional images captured by the cameras and the three-dimensional space needs to be determined.

[0109] S13. Based on the mapping relationship, the video streams collected by the multi-channel camera devices are projected onto the same plane, and image processing operations are performed on the video streams projected onto the same plane to obtain an overall monitoring video stream.

[0110] Specifically, when multiple cameras share their fields of view, the field of view (i.e., the captured video stream) of the corresponding camera device is projected onto the same plane for information sharing according to the mapping relationship expression of each camera device (i.e., the mapping relationship between the two-dimensional image and the three-dimensional space mentioned above). By performing image stitching and fusion operations on the video streams projected onto the same plane, the images corresponding to the same time point in all the video streams are combined into an overall field of view, i.e., the overall monitoring video stream, for subsequent algorithm judgment, thereby effectively eliminating the blind spots in the field of view caused by the stacking of objects during handover and improving the accuracy of the algorithm.

[0111] S14. Obtain a pre-configured detection model.

[0112] In this embodiment, since the computing power of the edge box is relatively weak, the model should not be too large. In order to improve the performance of the model while reducing the size of the model, a large model is used to guide the learning of the small model, and the knowledge of the large model is passed to the small model, thereby achieving higher operating efficiency and performance.

[0113] When implementing a large model to guide the learning of a small model, a large and powerful Transformer-based target detection model is used as a guidance model, which is the preset large model in the embodiment of the present invention. A target detection model based on a lightweight network MobileNet is used as a target model, which is the detection model in the embodiment of the present invention.

[0114] The detection model is obtained through training. The model training process includes four parts: data collection and processing, data labeling, model training, and model testing and format conversion.

[0115] During data collection and processing, data on the handover of items (such as cash, trunks, etc.) between external personnel and internal personnel is obtained. The data in this embodiment may be pictures or video frames in a video stream.

[0116] When annotating data, we filter and remove duplicates by calculating the similarity of all data. When removing duplicates, if the image similarity is greater than a certain threshold, the images are considered similar and the subsequent similar images are deleted. The deduplicated data is used as the model training set; the labels of the data in the training set are set to external personnel, internal personnel, and objects. In addition, to improve the robustness of the model, public personnel data and irrelevant scene data can also be integrated as training data.

[0117] After obtaining the training set, the training set is used to train the target detection model based on the Transformer structure. After the training, a batch of sample data is obtained and the sample data is input into the target detection model based on the Transformer structure to obtain the target position output by the model and the category prediction result corresponding to the target position.

[0118] For example, a sample data is input into the target detection model based on the Transformer structure. The output result of the model is the labeling result of the sample data. In the labeling result, people and objects are framed in the form of rectangular frames. The rectangular frame is the target position. For each target position, the score of the object inside it is marked as an external person, an internal person, and an object. For example, a target position has a score of 1 as an external person, 2 points as an internal person, and 4 points as an object. Since the score as an object is the highest, the object at the target position is considered to be an object.

[0119] After obtaining the result of each sample data being input into the Transformer, the detection model can be trained using these sample data and the output results of the Transformer.

[0120] Specifically, the training samples used in the detection model during training are: sample data and the probability distribution corresponding to the sample data; the probability distribution includes: a first probability distribution and a second probability distribution, the first probability distribution is: the probability distribution corresponding to the target position obtained by inputting the sample data into a preset large model, for the target position, the position information can be converted into a probability distribution using techniques such as coordinate regression.

[0121] The second probability distribution is: the probability distribution corresponding to the category prediction result obtained by inputting the sample data into the preset large model, which can be specifically processed by the softmax function to obtain the probability distribution corresponding to the category prediction result, specifically converting the original score value into a probability value. For example, if a target location has a score of 1 as an outsider, 2 as an insider, and 4 as an object, it is converted into a probability distribution of 1 / (1+2++4), 2 / (1+2++4), and 4 / (1+2++4), that is, converted into 1 / 7, 2 / 7, and 4 / 7.

[0122] The first probability distribution above represents the probability distribution of the target position, and the second probability distribution above can be a probability vector, in which each element represents the probability of a category.

[0123] The first probability distribution and the second probability distribution can be used as soft labels. The constructed soft labels are used as target values ​​of the sample data of the detection model. The model is trained using supervised learning, so that the model can reproduce the prediction results of the guidance model as much as possible while maintaining the simplicity and efficiency of the model.

[0124] In order to improve the accuracy of model training, the detection model uses a preset parameter optimization method to optimize the configuration of model parameters during training. The preset parameter optimization method can be a Bayesian optimizer to improve the accuracy of model parameter configuration, thereby improving the accuracy of model detection.

[0125] S15. Determine the personnel detection result and the object detection result corresponding to the overall monitoring video stream by using the detection model.

[0126] Specifically, after the detection model training is completed, the model can be put into use online, and the overall monitoring video stream is subsequently used as the input of the detection model to obtain the person detection results and object detection results corresponding to the overall monitoring video stream.

[0127] The personnel detection result includes the target position and type of personnel, such as detecting four external personnel and two internal personnel, and marking the target position of the personnel with a rectangular frame.

[0128] The object detection result includes the target location of the object and the object type. For example, if cash is detected, the cash is marked with a rectangular frame.

[0129] S16. Determine the vault handover monitoring result according to the personnel detection result and the item detection result.

[0130] Specifically, after obtaining the personnel detection results and the item detection results, when the item detection result is cash or trunk, the corresponding number of personnel can be checked to determine whether the number is in compliance and the vault handover monitoring result can be obtained. If the number of external personnel in the entire handover area is less than the specified number, an early warning will be issued; if the number of internal personnel in the entire handover area is less than the specified number, an early warning will be issued.

[0131] In this embodiment, the safe handover monitoring area is configured with multiple cameras that capture different perspectives to capture video streams from different perspectives of the safe handover monitoring area, thereby achieving multi-angle monitoring and avoiding the problem of limited monitoring range caused by obstruction of objects.

[0132] In addition, in the present application, the mapping relationship between the two-dimensional image captured by the camera device and the three-dimensional space is determined, and based on the mapping relationship, the video streams captured by the multi-channel camera device are projected onto the same plane, and the video streams projected onto the same plane are subjected to image processing operations to obtain an overall monitoring video stream, so that the complete video stream of the required monitoring area can be fully viewed in the same monitoring video stream, so that the detection model can be used to determine the personnel detection results and the object detection results corresponding to the overall monitoring video stream, and the vault handover monitoring results can be determined based on the personnel detection results and the object detection results. In the embodiment of the present invention, since the video stream is used for monitoring operations in real time, real-time monitoring and real-time warning can be achieved; it can significantly reduce the risk control risks and labor costs of manual supervision of the scene, avoid the problem of omissions in manual monitoring, and achieve real-time warning to reduce time loss and solve the lag problem, thereby improving monitoring accuracy.

[0133] In addition, in the present application, the training samples used in the training of the detection model are: sample data and the probability distribution corresponding to the sample data; the probability distribution includes: a first probability distribution and a second probability distribution, the first probability distribution is: the probability distribution corresponding to the target position obtained by inputting the sample data into the preset large model; the second probability distribution is: the probability distribution corresponding to the category prediction result obtained by inputting the sample data into the preset large model. Since the training samples use the probability distribution corresponding to the output result of the preset large model, a small-volume model can be used to realize personnel detection and object detection, reducing the resource occupation of the model. In addition, in the present invention, when the detection model is trained, the preset parameter optimization method is used to optimize the configuration of the model parameters, thereby ensuring the accuracy of the parameters in the model and improving the accuracy of the model detection.

[0134] The above embodiment mentions determining the mapping relationship between the two-dimensional picture captured by the camera device and the three-dimensional space, and now the specific implementation thereof is introduced. Figure 5 , determining the mapping relationship between the two-dimensional picture captured by the camera device and the three-dimensional space may include:

[0135] S21. Using the multi-channel camera device to capture images of reference plates at different positions and in different postures, and obtaining the reference plate position coordinates when capturing the images.

[0136] Specifically, since the objects overlap and pile up, and their volume is large, the problem of blocking people during the handover may be solved. To solve this problem, all fixed cameras with different fields of view and angles in the area are first aimed at the same checkered plate to shoot, and then the feature points of all checkered plate images are extracted and the projection transformation relationship of each camera is established based on projection geometry. Then, these projection transformations are combined to carry out multi-camera linkage and share information to project the image onto a common plane, and an overall scene field of view is formed through image stitching and fusion. Finally, the categories and item status of all people in the handover area are identified based on the overall field of view, and it is determined whether the number of two types of people in the handover is normal.

[0137] In order to achieve the above-mentioned fusion, a reference plate, such as a checkered plate reference plate, may be photographed. During the shooting, in order to achieve better fusion, the images may be shot from multiple postures and positions.

[0138] After determining the installation position of the multi-channel camera device, the installation operation is carried out. After the installation is completed, since the angle and position of each camera device are fixed, it is necessary to place the chessboard-shaped reference board in different positions in various postures. For example, the reference board is placed at an angle at position A, the reference board is placed flat at position B, and the reference board is placed vertically at position C. In different positions and postures, the multi-channel camera device is used to collect pictures of the reference board. For one position and one posture, each camera device will collect a picture of the reference board. The collected pictures of the reference board are used Figure 6 The discrete pixel coordinates shown in need to be converted into three-dimensional space to achieve spatial fusion. Figure 6 The projection relationship between the discrete pixel coordinates in the image is mathematically modeled, which involves the projection transformation of four coordinate planes. Specifically, the discrete pixel coordinates (u, v) are first converted to imaging coordinates (x, y), and then converted to camera coordinates (X C ,Y C ,Z C ), and finally converted to three-dimensional space coordinates (X W ,Y W ,Z W ), the specific conversion process is realized through the mapping relationship expression.

[0139] When taking a picture of the reference plate, the position coordinates (X W ,Y W ,Z W ). The four corners (X W ,Y W ,Z W ) are the reference plate position coordinates in the embodiment of the present invention.

[0140] The position and posture are subsequently adjusted, and a camera device is used to capture and acquire reference board images of multiple positions and postures for each camera.

[0141] S22, extracting pixel coordinates of corner points of the image.

[0142] Specifically, after obtaining the image of the reference board, the pixel coordinates of the corner points of the reference board image are extracted. Specifically, all the corner grid points of the chessboard are extracted from the reference board image taken by each camera device, and the pixel coordinates (u, v) of the corner points of each grid point are recorded.

[0143] S23, using the camera parameters of the multi-channel camera device, according to the corner point pixel coordinates and the reference plate position coordinates, calculating mapping parameters.

[0144] Specifically, the mapping relationship between the two-dimensional picture and the three-dimensional space, that is, the mapping relationship expression H, is:

[0145]

[0146] Among them, the camera parameters f, f of each camera device x 、f y is known in advance, x, y are imaging coordinates, u, v are discrete pixel coordinates, Z c is the coordinate of the point on the Z axis in the camera coordinate system, that is, the depth of the point. u0 is the u coordinate of the origin of the image coordinate axis on the pixel coordinate axis, and v0 is the v coordinate of the origin of the image coordinate axis on the pixel coordinate axis. They are fixed after leaving the factory. The mapping parameters [R, T] are the parameters that need to be calculated.

[0147] After acquiring images of different positions and postures, the above mapping relationship expression H is used to perform conversion operations to obtain the required R and T.

[0148] S24: Determine a mapping relationship between the two-dimensional image captured by the camera device and the three-dimensional space corresponding to the mapping parameters.

[0149] After obtaining R and T, substitute the mapping relationship expression H into the mapping relationship expression to obtain the final mapping relationship between the two-dimensional image of each camera device and the three-dimensional space. After the video stream is acquired by the camera device, the video frames at the same time point in each video stream are converted to the same plane using the mapping relationship. After the subsequent image stitching and fusion operations are performed, the overall monitoring video stream is obtained. The schematic diagram of image stitching and fusion can be referred to Figure 7 shown.

[0150] Since multiple cameras are used to collect video streams for the same monitoring area, the problems of being blocked by objects and the limited monitoring content of a single camera can be avoided during the splicing and fusion operations, thus achieving all-round monitoring.

[0151] In the embodiment of the present invention, by placing the reference board in multiple postures and positions and collecting corresponding pictures, the mapping parameters can be accurately calculated, and then the mapping relationship between the two-dimensional image and the three-dimensional space can be obtained. Through multi-camera field of view sharing, the field of view images of multiple cameras are projected onto the same plane according to the mapping relationship expression of each camera for information sharing. Through image stitching and fusion, these images are combined into an overall field of view image for subsequent algorithm judgment, effectively eliminating the field of view blind spots caused by stacking of objects during handover, and improving the accuracy of the algorithm.

[0152] Another embodiment of the present invention provides a process of optimizing the configuration of model parameters using a preset parameter optimization method, which may specifically include:

[0153] 1) Determine the weighted objective function and the hyperparameter space of the hyperparameters.

[0154] Specifically, when determining the weighted objective function, since the detection model of the present invention must accurately identify the coordinates and positions of people and objects, as well as the categories of people and objects, the weighted index of mean Average Precision (mAP) and recognition accuracy is used as the optimal weighted objective function.

[0155] When confirming the hyperparameter space, determine the learning rate, data enhancement strategy, and structural parameters of the network that need to be optimized. These parameters are hyperparameters. It is necessary to confirm the value range of these parameters and use them as the hyperparameter space.

[0156] 2) Select the Gaussian process surrogate model.

[0157] In this embodiment, when selecting the Gaussian process proxy model, for the personnel category model, the Gaussian process is selected as the proxy model to fit the weighted objective function, which is closer to the actual feature distribution and increases the selection space.

[0158] 3) Determine hyperparameter configuration information using the Gaussian process proxy model and iteration strategy.

[0159] Specifically, the iterative optimization process begins, and the hyperparameter configuration information for the next iteration is selected according to the Gaussian process surrogate model and the iteration strategy.

[0160] Among them, in the iterative strategy, the direction of the next parameter iteration is selected through the expected improvement strategy, and the parameters are improved and iterated in the direction with the smallest coordinate prediction error and the largest category recognition accuracy.

[0161] 4) According to the hyperparameter configuration information, the hyperparameter is optimized in the hyperparameter space, and after the optimization operation is completed, the function value of the weighted objective function is calculated.

[0162] Specifically, after obtaining the hyperparameter configuration information, the hyperparameter configuration information is used to perform an optimization operation on the hyperparameter in the hyperparameter space to obtain the hyperparameter in this iterative optimization process, and the hyperparameter is used to calculate the function value of the weighted objective function in this iterative optimization process.

[0163] 5) If the function value of the weighted objective function does not converge to within a preset threshold range, update the Gaussian process proxy model based on the function value of the weighted objective function, and return to the step of determining the hyperparameter configuration information using the Gaussian process proxy model and the iteration strategy.

[0164] Specifically, if the function value of the weighted objective function does not converge to within the preset threshold range, it means that the current iterative operation has not reached the optimal state. At this time, the Gaussian process proxy model can be updated based on the true function value of the weighted objective function, and then the next iteration is performed. That is, the Gaussian process proxy model and the iteration strategy are used to determine the hyperparameter configuration information, and the operation is executed sequentially until the function value of the new weighted objective function is obtained. Then, it is judged whether it converges. If not, the next iteration is performed.

[0165] 6) If the function value of the weighted objective function converges to within a preset threshold range, the obtained hyperparameters are used as model parameters of the detection model.

[0166] If the function value of the weighted objective function converges to within the preset threshold range, it means that the iteration has converged and the optimal hyperparameter configuration information has been obtained. The iteration operation can be stopped. At this time, the obtained hyperparameters are used as model parameters of the detection model for model training.

[0167] In this embodiment, when training the detection model, a Bayesian optimizer is used to optimize the configuration of model parameters to obtain the optimal parameter configuration of the detection model, thereby improving the accuracy of model recognition.

[0168] Another embodiment of the present invention provides a process for using the detection model to determine a person detection result and an object detection result corresponding to the overall surveillance video stream, which may include:

[0169] 1) Obtain the current bit rate of multiple camera devices.

[0170] In actual applications, when using detection models to perform personnel detection and object detection, since there are a large number of video frames in the video stream, if each video frame is detected, the detection task will be large. Therefore, in the embodiment of the present invention, a frame extraction operation is adopted to extract some video frames, and detection operations are only performed on the extracted video frames.

[0171] During the extraction process, it is necessary to determine the extraction frequency of the video frames. When determining the extraction frequency, the bit rate of the camera device, the current network conditions, and the results of the vault handover monitoring in the previous detection cycle may be considered.

[0172] Among them, the higher the bit rate of the camera device, the higher the clarity of the video stream it generates, which will cause the amount of data carried by the video frames in the video stream to increase. Therefore, when the bit rate is high, the value of the extraction frequency can be reduced, and when the bit rate is low, the value of the extraction frequency can be increased.

[0173] The bit rate of the camera device is an inherent parameter of the camera device and can be directly obtained.

[0174] Regarding the network situation, if the current network situation is good, such as a fast network speed, the video stream transmitted from the camera device to the edge computing device will be smoother, with lower latency and better continuity. At this time, the transmitted video stream is a real-time video stream, and fewer video frames can be extracted. If the current network situation is poor, such as a low network speed, the video stream transmitted from the camera device to the edge computing device will be less smooth, with higher latency and poorer continuity. At this time, the transmitted video stream may not be a real-time video stream, so more video frames can be extracted to obtain more information.

[0175] For the safe handover monitoring results of the previous detection cycle, if no people or items are detected, it means that the current scene is not a handover scene. You can extract fewer video frames, that is, reduce the value of the extraction frequency. If people or items are detected, it means that the current scene is a handover scene. In order to achieve real-time monitoring, you can extract more video frames, that is, increase the value of the extraction frequency.

[0176] 2) Get current network data.

[0177] In this embodiment, the network data can be determined by collecting data such as signal strength.

[0178] 3) Obtain the vault handover monitoring results of the previous inspection cycle.

[0179] In this embodiment, the last detection cycle is the safe handover monitoring result of the last extracted video frame detected, specifically including the personnel detection result and the object detection result.

[0180] 4) Determine the video stream frame extraction rule according to the bit rate of the multi-channel camera device, the current network data and the safe handover monitoring result detected in the previous detection cycle.

[0181] Specifically, after obtaining the bit rate of the multi-channel camera device, the current network data and the safe handover monitoring results detected in the previous detection cycle, the extraction frequency corresponding to each parameter (i.e., the bit rate of the multi-channel camera device, the current network data and the safe handover monitoring results detected in the previous detection cycle) can be determined respectively, and then the extraction frequency is weightedly summed to obtain the extraction frequency in the video stream frame extraction rule.

[0182] 5) According to the video stream frame extraction rule, extract the target video frame from the overall monitoring video stream, and use the detection model to determine the person detection result and the object detection result corresponding to the target video frame.

[0183] In this embodiment, after determining the extraction frequency in the video stream frame extraction rule, the extraction frequency is used to extract the target video frame from the overall monitoring video stream. For example, if the extraction frequency is one frame out of five frames, then one video frame can be extracted from five video frames, and the target video frame can be processed using the model to obtain personnel detection results and object detection results.

[0184] After the detection is completed, the next extraction frequency is calculated and subsequent frame extraction detection operations are performed until there are no people or objects in the video stream collected by the camera device.

[0185] In this embodiment, the extraction frequency can be adjusted according to the real-time scenario to achieve real-time adjustment of video frame extraction, so that the video frame extraction not only meets the current bit rate and network conditions, but also adapts to the current handover scenario. On the premise of meeting the needs of personnel and object detection, the video frame processing volume is reduced, thereby reducing resource consumption.

[0186] In another implementation of the present invention, step S16 includes the following steps:

[0187] 1) Determine whether the object detection result corresponding to the target video frame is a preset object; if so, calculate the total number of different types of people according to the person detection result corresponding to the target video frame;

[0188] Among them, the preset items in the present invention refer to cash or a trunk. By detecting whether it is a preset item, it is also determined whether the current scene is a vault handover scene.

[0189] Specifically, the object detection result corresponding to the target video frame is obtained to determine whether the content is cash or a trunk. If so, it means that the current scene is a vault handover, and the total number of different types of personnel is further determined.

[0190] When determining the total number of personnel, the total number of personnel of different types in the personnel detection results may be counted, such as the total number of internal personnel and the total number of external personnel.

[0191] 2) Calculate the average of the total number of different types of personnel.

[0192] In order to avoid the problem of inaccurate detection results in a certain video frame, such as not detecting a person squatting, in this embodiment, the total number of people detected in the previous video frame or the previous few video frames can be averaged to obtain the total number of internal personnel and the total number of external personnel.

[0193] 3) If the average total number of different types of personnel is greater than the preset threshold, the vault handover monitoring result is determined to be a first indicator representing compliance; if the average total number of at least one type of personnel is not greater than the preset threshold, the vault handover monitoring result is determined to be a second indicator representing violation.

[0194] Specifically, if the average total number of different types of personnel is greater than the preset threshold, it means that the current number of personnel meets the requirements, and the vault handover monitoring result is the first mark representing compliance. The first mark can be a compliant text or a mark representing compliance, such as 0.

[0195] If the average total number of at least one type of personnel is not greater than the preset threshold, indicating that the current number of personnel does not meet the requirement, the vault handover monitoring result is the second mark representing the violation, which can be a text of the violation or a mark representing the violation. If it is 1, an early warning operation is performed. That is, if the number of external personnel in the entire handover area is less than the specified number, an early warning is issued, and if the number of internal personnel in the entire handover area is less than the specified number, an early warning is issued.

[0196] In this embodiment, the detection model monitors the designated handover area through target detection. When a trunk item of a designated shape and color is detected in the handover area, the number and category of personnel are detected to determine whether the number and category of personnel on both sides of the handover meet the requirements. In order to reduce the impact of accidental misidentification as a result, the detection results of the continuous time series are counted and averaged. If the average exceeds the preset threshold, it is determined that it does not meet the requirements and a reminder message is sent for warning.

[0197] like Figure 8 As shown, the real-time early warning system for treasury in and out in the embodiment of the present invention mainly includes three modules: model training, algorithm development and deployment and operation.

[0198] The model training module includes four functions: data collection and processing, data labeling, model training, and model testing and format conversion.

[0199] Data collection and processing: Obtain data on item handover between external and internal personnel.

[0200] Data labeling: Filter and deduplicate data by calculating the similarity of all data as the model training set (when the image similarity is greater than a certain threshold, the images are considered similar and subsequent similar images are deleted); set the model labels to external personnel, internal personnel and objects.

[0201] Model training: Since edge boxes have weak computing power, the model cannot be too large. In order to improve model performance while reducing model size, a large model is used to guide the learning of a small model, and the knowledge of the large model is passed to the small model, thereby achieving higher operating efficiency and performance.

[0202] The large model guides the learning of the small model: a large and powerful Transformer-based target detection model is used as the guidance model, and a lightweight network MobileNet-based target detection model is used as the target model. The labeled data is trained with the guidance model to obtain the target location and category prediction results for each sample. For location information, coordinate regression and other techniques are usually used to convert the location information into a probability distribution representing the location of the target, and the prediction results after probabilistic processing are used as soft labels. For category prediction results, the soft label of each sample is a probability vector, in which each element represents the probability of a category. The constructed soft label is used as the target value of the target model, and the target model is trained using supervised learning. The purpose of the target model is to reproduce the prediction results of the guidance model as much as possible while keeping the model simple and efficient.

[0203] Model testing and format conversion: Convert the model into the cross-platform ONNX (Open Neural Network Exchange) format so that subsequent deployment can be carried out according to the architecture platform of the specific edge device to perform corresponding model conversion and tailoring, making full use of the computing performance of different hardware platforms.

[0204] The deployment and operation include four modules: stream data processing layer, reasoning layer, logic processing layer and warning push layer.

[0205] Vault surveillance camera channel: Multiple cameras at different angles and directions capture the vault handover area in real time, and input surveillance video data from multiple cameras around the vault handover area into local edge computing devices, such as edge boxes, in real time.

[0206] Stream data processing layer: decodes, extracts frames, and performs data preprocessing on real-time video streams, converts video stream data into continuous-time image sequences, and then preprocesses the image data to obtain standard image data that meets the model input requirements.

[0207] Reasoning layer: The model deployment and reasoning layer performs model reasoning for vault items and personnel categories, and obtains the corresponding detection frame coordinates, size, labels and other reasoning results. In order to adapt to different edge hardware architectures at low cost, a modular design is used to decompose the reasoning layer into a model conversion and optimization module, a hardware acceleration module, a dynamic load adjustment module and a platform compatibility module, to achieve program decoupling and modular packaging, forming a device-dependent part and a device-general part to improve maintainability and flexibility.

[0208] Among them, the model conversion and optimization module: converts the general ONNX model into a model type suitable for the device by calling the API (Application Programming Interface) interface of the corresponding device manufacturer, and then performs specific model optimizations such as quantization, compression and cropping on the converted model to better adapt to and utilize the hardware structure and performance of the device.

[0209] Hardware acceleration module: adjust the model's hyperparameters and configuration according to the device's computing power standards and performance configuration, and select the optimal hardware accelerator for inference acceleration to give full play to the device's computing power. This module, along with the model conversion and optimization model, are all highly device-dependent parts and need to be adapted separately for different device manufacturers.

[0210] Dynamic load adjustment module: Since the configured camera bit rate and network conditions are changing, this module is designed to dynamically adjust the configuration of the hardware acceleration module according to the current device load to achieve optimal performance and resource utilization.

[0211] Platform compatibility module: ensures that the inference layer can run on different operating systems and software platforms, and can be well integrated with the stream data processing layer and the logic processing layer. This module and the dynamic load adjustment module are common parts of the device and can be applied to devices with different hardware architectures, reducing the burden of development and adaptation.

[0212] Logical processing layer: algorithm deployment and operation layer. Logically process the object detection results and personnel detection results output by the reasoning layer to determine whether there are any violations in the number and type of personnel during the handover. Specifically, when a trunk item of a specified shape and color is detected in the handover area, determine whether the number and type of personnel in the area meet the specified requirements. If not, there is a violation. According to the reasoning results, draw the vault items, staff and related violation detection boxes on the monitoring screen to form a violation warning map.

[0213] Early warning push layer: The early warning push layer packages and pushes the violation warning map, the image, and related alarm information such as time, location, and violation event number to the cloud platform.

[0214] Cloud platform: Receives, processes and stores alarm information from edge boxes, and sends the alarm information to management personnel via SMS, phone calls, corporate WeChat and other channels for timely processing.

[0215] In this embodiment, a system for real-time analysis and early warning of the vault handover area is established through machine vision and edge computing technology. Through real-time analysis of the vault items and personnel categories in the monitoring screen, it is determined whether the compliance requirements are met and an early warning is issued, and then the early warning information is pushed to the cloud platform, which processes the information and pushes notifications to relevant management personnel to achieve normalized real-time monitoring of the vault door area and reduce the manpower burden of bank security.

[0216] In addition, the present invention can cover every vault handover event and realize comprehensive coverage inspection. In addition, the present invention can conduct real-time inspection and early warning around the clock, reduce the cost and time loss of post-event review, and save a lot of manpower cost and time.

[0217] A safe handover monitoring method provided by an embodiment of the present application is introduced above, and a device for executing the safe handover monitoring method will be introduced below.

[0218] See also Fig. 9 , Fig. 9 This is a schematic diagram of the structure of a vault handover monitoring device provided in an embodiment of the present application. Fig. 9 As shown, the safe handover monitoring device includes:

[0219] The video stream acquisition module 11 is used to acquire the video stream collected by the multi-channel camera devices in the safe handover monitoring area, wherein the safe handover monitoring area is configured with multiple camera devices collecting different viewing angles to collect the video streams of different viewing angles of the safe handover monitoring area;

[0220] A relationship determination module 12, used to determine a mapping relationship between the two-dimensional image captured by the camera device and the three-dimensional space;

[0221] An integration module 13 is used to project the video streams collected by the multi-channel camera devices onto the same plane based on the mapping relationship, and perform image processing operations on the video streams projected onto the same plane to obtain an overall monitoring video stream;

[0222] The model acquisition module 14 is used to acquire a pre-configured detection model. The training samples used in the training of the detection model are: sample data and probability distribution corresponding to the sample data; the probability distribution includes: a first probability distribution and a second probability distribution. The first probability distribution is: the probability distribution corresponding to the target position obtained by inputting the sample data into the preset large model; the second probability distribution is: the probability distribution corresponding to the category prediction result obtained by inputting the sample data into the preset large model; when the detection model is trained, the preset parameter optimization method is used to optimize the configuration of model parameters;

[0223] A detection module 15, configured to use the detection model to determine a person detection result and an object detection result corresponding to the overall monitoring video stream; the person detection result includes a target position and a type of person; the object detection result includes a target position and a type of object;

[0224] The monitoring determination module 16 is used to determine the vault handover monitoring result according to the personnel detection result and the item detection result.

[0225] In a possible implementation, the relationship determination module 12 includes:

[0226] A coordinate acquisition submodule, used to use the multi-channel camera device to capture images of reference plates at different positions and in different postures, and to obtain the reference plate position coordinates when the reference plates are capturing images;

[0227] A coordinate extraction submodule, used to extract the pixel coordinates of the corner points of the image;

[0228] A parameter calculation submodule, used to calculate mapping parameters according to the corner point pixel coordinates and the reference plate position coordinates using the camera parameters of the multi-channel camera device;

[0229] The relationship determination submodule is used to determine the mapping relationship between the two-dimensional picture captured by the camera device and the three-dimensional space corresponding to the mapping parameters.

[0230] In a possible implementation, the integration module 13 includes:

[0231] The integration submodule is used to perform image stitching and fusion operations on the video streams projected onto the same plane to obtain the overall monitoring video stream.

[0232] In a possible implementation, an optimization configuration module is further included, and the optimization configuration module includes:

[0233] A data determination submodule, for determining a weighted objective function and determining a hyperparameter space of hyperparameters;

[0234] Model selection submodule, used to select Gaussian process surrogate model;

[0235] A configuration determination submodule, used to determine hyperparameter configuration information using the Gaussian process proxy model and iteration strategy;

[0236] An optimization submodule, configured to optimize the hyperparameters in the hyperparameter space according to the hyperparameter configuration information, and calculate the function value of the weighted objective function after the optimization operation is completed;

[0237] An updating module, configured to update the Gaussian process proxy model based on the function value of the weighted objective function if the function value of the weighted objective function does not converge to within a preset threshold range;

[0238] The configuration determination submodule is further used to determine hyperparameter configuration information by using the Gaussian process proxy model and an iteration strategy after the Gaussian process proxy model is updated;

[0239] The parameter determination submodule is used to use the obtained hyperparameters as model parameters of the detection model if the function value of the weighted objective function converges to within a preset threshold range.

[0240] In a possible implementation, the detection module 15 includes:

[0241] The bit rate acquisition submodule is used to acquire the bit rate of the current multi-channel camera device;

[0242] The network acquisition submodule is used to obtain current network data;

[0243] The monitoring result acquisition submodule is used to obtain the vault handover monitoring results detected in the previous detection cycle;

[0244] A frame extraction determination submodule, used to determine a video stream frame extraction rule according to the bit rate of the multi-channel camera device, the current network data and the vault handover monitoring result detected in the previous detection cycle;

[0245] The detection submodule is used to extract the target video frame from the overall monitoring video stream according to the video stream frame extraction rule, and use the detection model to determine the person detection result and the object detection result corresponding to the target video frame.

[0246] In a possible implementation, the monitoring determination module 16 includes:

[0247] A judgment submodule, used to judge whether the object detection result corresponding to the target video frame is a preset object;

[0248] A quantity calculation submodule, for calculating the total quantity of different types of personnel according to the personnel detection result corresponding to the target video frame;

[0249] The average value calculation submodule is used to calculate the average value of the total number of different types of personnel;

[0250] The result determination submodule is used to determine that the vault handover monitoring result is a first identification indicating compliance if the average value of the total number of different types of personnel is greater than a preset threshold; if the average value of the total number of at least one type of personnel is not greater than the preset threshold, determine that the vault handover monitoring result is a second identification indicating violation.

[0251] In this embodiment, the vault handover monitoring area is equipped with multiple cameras that collect different viewing angles to collect video streams from different viewing angles of the vault handover monitoring area, thereby realizing multi-angle monitoring, and avoiding the problem of limited monitoring range caused by object occlusion. In addition, in this application, the mapping relationship between the two-dimensional picture collected by the camera and the three-dimensional space is determined, and based on the mapping relationship, the video streams collected by the multiple camera devices are projected onto the same plane, and the video streams projected onto the same plane are subjected to image processing operations to obtain an overall monitoring video stream, so that the complete video stream of the required monitoring area can be fully viewed in the same monitoring video stream, so that the detection model can be used to determine the personnel detection results and the item detection results corresponding to the overall monitoring video stream, and the vault handover monitoring results can be determined according to the personnel detection results and the item detection results. In the embodiment of the present invention, since the video stream is used in real time for monitoring operations, the problem of omissions in manual monitoring can be avoided, and the monitoring accuracy can be improved. In addition, in the present application, the training samples used in the training of the detection model are: sample data and the probability distribution corresponding to the sample data; the probability distribution includes: a first probability distribution and a second probability distribution, the first probability distribution is: the probability distribution corresponding to the target position obtained by inputting the sample data into the preset large model; the second probability distribution is: the probability distribution corresponding to the category prediction result obtained by inputting the sample data into the preset large model. Since the training samples use the probability distribution corresponding to the output result of the preset large model, a small-volume model can be used to realize personnel detection and object detection, reducing the resource occupation of the model. In addition, in the present invention, when the detection model is trained, the preset parameter optimization method is used to optimize the configuration of the model parameters, thereby ensuring the accuracy of the parameters in the model and improving the accuracy of the model detection.

[0252] It should be noted that, for the working process of each module and sub-module in this embodiment, please refer to the corresponding description in the above embodiment, which will not be repeated here.

[0253] An embodiment of the present application also provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0254] The memory is used to store computer programs;

[0255] The processor is used to execute the computer program so that the electronic device can implement the above-mentioned safe handover monitoring method.

[0256] refer to Fig.10 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic device in the embodiments of the present application may include but is not limited to fixed terminals such as mobile phones, notebook computers, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Fig.10 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0257] like Fig.10 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0258] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Fig.10 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0259] Also provided in an embodiment of the present application is a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any of the safe handover monitoring methods provided in the embodiments of the present application.

[0260] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the safe handover monitoring methods provided in the embodiments of the present application.

[0261] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0262] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0263] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0264] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A vault handover monitoring method, characterized in that: include: Acquire video streams collected by multiple camera devices in the safe handover monitoring area, wherein the safe handover monitoring area is configured with multiple camera devices collecting different viewing angles to collect video streams from different viewing angles of the safe handover monitoring area; Determining a mapping relationship between the two-dimensional image captured by the camera device and the three-dimensional space; Based on the mapping relationship, the video streams collected by the multi-channel camera devices are projected onto the same plane, and image processing operations are performed on the video streams projected onto the same plane to obtain an overall monitoring video stream; A pre-configured detection model is obtained, wherein the training samples used by the detection model during training are: sample data and probability distribution corresponding to the sample data; the probability distribution includes: a first probability distribution and a second probability distribution, wherein the first probability distribution is: a probability distribution corresponding to a target position obtained by inputting the sample data into a preset large model; the second probability distribution is: a probability distribution corresponding to a category prediction result obtained by inputting the sample data into the preset large model; during training of the detection model, the preset parameter optimization method is used to optimize the configuration of model parameters; Determine the personnel detection result and the object detection result corresponding to the overall monitoring video stream by using the detection model; the personnel detection result includes the target position and the type of the personnel; the object detection result includes the target position and the type of the object; The vault handover monitoring result is determined based on the personnel detection result and the item detection result.

2. The method for monitoring the handover of a safe according to claim 1, characterized in that: Determining a mapping relationship between a two-dimensional image captured by the camera device and a three-dimensional space includes: Using the multi-channel camera device to collect pictures of the reference plate at different positions and in different postures, and obtaining the reference plate position coordinates of the reference plate when the pictures are collected; Extracting pixel coordinates of corner points of the image; Calculating mapping parameters according to the corner point pixel coordinates and the reference plate position coordinates using the camera parameters of the multi-channel camera device; A mapping relationship between the two-dimensional picture captured by the camera device and the three-dimensional space corresponding to the mapping parameters is determined.

3. The method for monitoring the handover of a safe according to claim 1, characterized in that: Perform image processing operations on the video stream projected onto the same plane to obtain the overall monitoring video stream, including: Perform image stitching and fusion operations on the video streams projected onto the same plane to obtain the overall monitoring video stream.

4. The method for monitoring the handover of a safe according to claim 1, characterized in that: The process of optimizing the model parameters using the preset parameter optimization method includes: Determining a weighted objective function and determining a hyperparameter space of hyperparameters; Select the Gaussian process surrogate model; Determine hyperparameter configuration information using the Gaussian process proxy model and iteration strategy; According to the hyperparameter configuration information, the hyperparameter is optimized in the hyperparameter space, and after the optimization operation is completed, the function value of the weighted objective function is calculated; If the function value of the weighted objective function does not converge to within a preset threshold range, updating the Gaussian process proxy model based on the function value of the weighted objective function, and returning to the step of determining the hyperparameter configuration information using the Gaussian process proxy model and the iteration strategy; If the function value of the weighted objective function converges to within a preset threshold range, the obtained hyperparameters are used as model parameters of the detection model.

5. The method for monitoring the handover of a safe according to claim 1, characterized in that: Using the detection model, determining the person detection result and the object detection result corresponding to the overall monitoring video stream includes: Get the current bit rate of multiple camera devices; Get current network data; Obtain the vault handover monitoring results of the previous inspection cycle; Determine a video stream frame extraction rule according to the bit rate of the multi-channel camera device, the current network data, and the vault handover monitoring result detected in the previous detection cycle; According to the video stream frame extraction rule, a target video frame is extracted from the overall monitoring video stream, and the detection model is used to determine the person detection result and the object detection result corresponding to the target video frame.

6. The method for monitoring the handover of a safe according to claim 5, characterized in that: According to the personnel detection results and the item detection results, the vault handover monitoring results are determined, including: Determine whether the object detection result corresponding to the target video frame is a preset object; If yes, calculate the total number of different types of people according to the person detection result corresponding to the target video frame; Calculate the average of the total number of different types of personnel; If the average total number of different types of personnel is greater than the preset threshold, the vault handover monitoring result is determined to be the first indicator of compliance; If the average value of the total number of at least one type of personnel is not greater than a preset threshold, the safe handover monitoring result is determined to be a second indicator representing a violation.

7. A safe handover monitoring device, characterized in that: include: A video stream acquisition module is used to acquire video streams collected by multiple camera devices in the safe handover monitoring area, wherein the safe handover monitoring area is configured with multiple camera devices that collect different viewing angles to collect video streams from different viewing angles of the safe handover monitoring area; A relationship determination module, used to determine the mapping relationship between the two-dimensional picture captured by the camera device and the three-dimensional space; An integration module, used for projecting the video streams collected by the multi-channel camera devices onto the same plane based on the mapping relationship, performing image processing operations on the video streams projected onto the same plane, and obtaining an overall monitoring video stream; A model acquisition module is used to acquire a pre-configured detection model. The training samples used in the training of the detection model are: sample data and probability distribution corresponding to the sample data; the probability distribution includes: a first probability distribution and a second probability distribution. The first probability distribution is: the probability distribution corresponding to the target position obtained by inputting the sample data into a preset large model; the second probability distribution is: the probability distribution corresponding to the category prediction result obtained by inputting the sample data into the preset large model; when the detection model is trained, the preset parameter optimization method is used to optimize the configuration of model parameters; A detection module, used to determine the personnel detection result and the object detection result corresponding to the overall monitoring video stream by using the detection model; the personnel detection result includes the target position and the type of the personnel; the object detection result includes the target position and the type of the object; The monitoring determination module is used to determine the vault handover monitoring result based on the personnel detection result and the item detection result.

8. A computer program product, characterized in that It includes computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the safe handover monitoring method as described in any one of claims 1 to 6.

9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the safe handover monitoring method as described in any one of claims 1 to 6.

10. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the safe handover monitoring method as described in any one of claims 1 to 6.

Images