Abnormal behavior detection method and device, storage medium and electronic device

Abstract

The application discloses an abnormal behavior detection method and device, a storage medium and an electronic device, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring a target image to be detected, wherein the target image is an image corresponding to a target region, and the target region is a region where a financial institution is located; inputting the target image into a target behavior detection model for detection, and outputting a detection result of the target image, wherein the target behavior detection model is a model obtained by training an original behavior detection model based on visible light images, infrared images obtained in a historical process, and images formed by fusing the visible light images and the infrared images, and the detection result is used for representing whether there is an abnormal behavior in the target image. Through the application, the problem that the effect of detecting abnormal behaviors in bank outlets in the related art is poor is solved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a method and apparatus for detecting abnormal behavior, a storage medium, and an electronic device. Background Technology

[0002] Two-stage object detection algorithms based on deep learning and object detection algorithms based on candidate boxes consist of two parts: (1) selecting candidate regions, for which the commonly used algorithms are Selective Search and Edge Boxes. (2) using CNN (Convolutional Neural Networks) to extract features from the candidate regions for classification. The following mainly introduces representative object detection algorithms based on candidate boxes.

[0003] Currently, Selective Search + CNN (meaning convolutional neural networks use selective search algorithms for object detection) can be used instead of the traditional sliding window + handcrafted features. Furthermore, by designing the R-CNN model framework, it can be seen that CNN significantly improves object detection performance on PASCAL VOC 2007 (a standard dataset and benchmark for image classification and recognition) compared to traditional handcrafted features such as HOG (Histogram of Oriented Gradient). Therefore, deep learning algorithms are effective and efficient in the field of object detection. Moreover, there are currently two-stage object detection networks represented by SPP-Net (Spatial Pyramid Pooling Networks), Fast-RCNN, and Faster R-CNN.

[0004] Furthermore, the key to speed improvements in deep learning-based one-stage object detection algorithms and candidate box-based object detection algorithms lies in transforming the object detection problem into a multi-region classification problem, while fully utilizing the contextual information of the object within the entire image. To address this, a regression-based object detection algorithm has been proposed that is significantly faster than candidate box-based algorithms. Regression-based algorithms eliminate the candidate box generation stage, instead treating all locations in the image as potential objects and attempting to classify each region of interest. Representative algorithms include the YOLO series and the SSD series.

[0005] However, the relevant technology has the following drawbacks:

[0006] (1) It consumes a lot of manpower and resources and cannot guarantee monitoring efficiency. At the same time, it will cause various video surveillance systems to expose various defects to varying degrees, such as: long alarm response time, poor alarm accuracy, alarm lag, many false alarms and missed alarms, and difficulty in analyzing video data. Therefore, it reduces the security and practicality of the entire system.

[0007] (2) Currently, most mainstream target detection algorithms use visible light images for target recognition, but there are significant limitations to using visible light images for target recognition: performance is greatly affected by environmental conditions. Recognition based solely on visible light images or videos is easily affected by changes in lighting. In dimly lit environments or under non-uniform lighting conditions, the recognition rate will be greatly reduced.

[0008] There is currently no effective solution to the problem that related technologies are not very effective at detecting abnormal behavior in bank branches. Summary of the Invention

[0009] The main objective of this application is to provide a method, apparatus, storage medium, and electronic device for detecting abnormal behavior, in order to solve the problem of poor effectiveness in detecting abnormal behavior in bank branches in related technologies.

[0010] To achieve the above objectives, according to one aspect of this application, a method for detecting abnormal behavior is provided. The method includes: acquiring a target image to be detected, wherein the target image is an image corresponding to a target region, and the target region is the area where a financial institution is located; inputting the target image into a target behavior detection model for detection; and outputting a detection result for the target image, wherein the target behavior detection model is a model trained on an original behavior detection model based on visible light images, infrared images, and an image formed by fusing the visible light images and the infrared images acquired in a historical process; and the detection result is used to indicate whether abnormal behavior exists in the target image.

[0011] Furthermore, the target behavior detection model is obtained by: acquiring an image set, wherein the image set includes at least N visible light images, M infrared images, and S fused images, wherein the fused images are images obtained by fusing visible light images and infrared images, and N, M, and S are all positive integers; and using the image set to learn and train the original behavior detection model to obtain the target behavior detection model.

[0012] Further, acquiring the image set includes: acquiring a visible light video corresponding to the target region and an infrared video corresponding to the target region; performing frame extraction processing on the visible light video to obtain T visible light images, where T is greater than N and T is a positive integer; performing frame extraction processing on the infrared video to obtain K infrared images, where K is less than M and K is a positive integer; determining the image entropy of each visible light image and the image entropy of each infrared image; calculating the ratio between the image entropy of each visible light image and the image entropy of each infrared image based on the image entropy of each visible light image and the image entropy of each infrared image to obtain a first ratio set, wherein the first ratio set includes at least U first ratios, where U is a positive integer; and determining the image set based on the U first ratios.

[0013] Further, determining the image set based on the U first ratios includes: if there is a ratio among the U first ratios greater than a first preset threshold, obtaining the N visible light images from the T visible light images; if there is a ratio among the U first ratios less than a second preset threshold, determining the M infrared images based on the K infrared images, wherein the second preset threshold is less than the first preset threshold; if there is a ratio among the U first ratios greater than the second preset threshold and less than the first preset threshold, determining the S fused images based on the T visible light images and the K infrared images; and summarizing the N visible light images, the M infrared images, and the S fused images to obtain the image set.

[0014] Further, if any of the U first ratios is less than a second preset threshold, determining the M infrared images based on the K infrared images includes: if any of the U first ratios is less than the second preset threshold, obtaining X infrared images from the K infrared images, where X is less than K and X is a positive integer; preprocessing the X infrared images to obtain P preprocessed infrared images, where P is greater than X and P is a positive integer; and using the P preprocessed infrared images as the M infrared images.

[0015] Further, preprocessing the X infrared images to obtain P preprocessed infrared images includes: performing homomorphic filtering on the X infrared images to obtain X homomorphically filtered infrared images; performing histogram equalization on the X infrared images to obtain X histogram equalized infrared images; and summing the X homomorphically filtered infrared images and the X histogram equalized infrared images to obtain the P preprocessed infrared images.

[0016] Further, performing homomorphic filtering on the X infrared images to obtain X homomorphically filtered infrared images includes: performing logarithmic transform on the X infrared images to obtain X logarithmically transformed infrared images; performing Fourier transform on the X logarithmically transformed infrared images to obtain X Fourier transformed infrared images; filtering the X Fourier transformed infrared images to obtain X filtered infrared images; and performing inverse Fourier transform on the X filtered infrared images to obtain the X homomorphically filtered infrared images.

[0017] Further, performing histogram equalization on the X infrared images to obtain X histogram-equalized infrared images includes: obtaining the signal-to-noise ratio (SNR) of each of the X infrared images; obtaining the SNR of each sub-image in each infrared image; calculating the ratio between the SNR of each infrared image and the SNR of each sub-image to obtain a second ratio set, wherein the second ratio set includes at least V second ratios, where V is a positive integer; determining the target pixel value in each sub-image based on the second ratio set; removing the target pixel value from the pixel value of each sub-image to obtain a sub-image set, wherein the sub-image set includes at least Q sub-images, where Q is a positive integer; and performing histogram equalization on each of the Q sub-images to obtain the X histogram-equalized infrared images.

[0018] Further, if among the U first ratios there exists a ratio greater than the second preset threshold and less than the first preset threshold, determining the S fused images based on the T visible light images and the K infrared images includes: if among the U first ratios there exists a ratio greater than the second preset threshold and less than the first preset threshold, obtaining Y visible light images from the T visible light images, where Y is less than T and Y is a positive integer; obtaining R infrared images from the K infrared images, where R is less than K and R is a positive integer; and preprocessing the R infrared images to obtain H preprocessed infrared images. The preprocessing process includes at least: homomorphic filtering and histogram equalization, where H is greater than R and H is a positive integer; determining the image entropy of each visible light image in the Y visible light images and the image entropy of each infrared image in the R infrared images; determining the weights corresponding to the visible light images and the infrared images based on the image entropy of each visible light image in the Y visible light images and the image entropy of each infrared image in the R infrared images; and fusing the Y visible light images and the H preprocessed infrared images based on the weights corresponding to the visible light images and the infrared images to obtain the S fused images.

[0019] Furthermore, after inputting the target image into the target behavior detection model for detection and outputting the detection result of the target image, the method further includes: if the detection result indicates that there is abnormal behavior in the target image, then obtaining a probability value that the abnormal behavior is a target abnormal behavior, wherein the risk level of the target abnormal behavior is higher than a preset risk level threshold; determining whether the probability value is greater than a third preset threshold; if the probability value is greater than the third preset threshold, then sending a reminder message to the target object, wherein the reminder message is used to remind the target object to handle the abnormal behavior in the target image; if the probability value is not greater than the third preset threshold, then prohibiting the sending of the reminder message to the target object.

[0020] Furthermore, the process of using the image set to train the original behavior detection model to obtain the target behavior detection model includes: determining a first loss function and a second loss function; weighting the first loss function and the second loss function to obtain a target loss function; and combining the target loss function with the image set to train the original behavior detection model to obtain the target behavior detection model.

[0021] To achieve the above objectives, according to another aspect of this application, an abnormal behavior detection device is provided. The device includes: a first acquisition unit, configured to acquire a target image to be detected, wherein the target image is an image corresponding to a target region, and the target region is the area where a financial institution is located; and a first detection unit, configured to input the target image into a target behavior detection model for detection, and output a detection result for the target image, wherein the target behavior detection model is a model trained on an original behavior detection model based on visible light images, infrared images, and an image formed by fusing the visible light images and the infrared images acquired in a historical process, and the detection result is used to indicate whether abnormal behavior exists in the target image.

[0022] Furthermore, the target behavior detection model is obtained through the following means: a second acquisition unit, used to acquire an image set, wherein the image set includes at least N visible light images, M infrared images, and S fused images, wherein the fused images are images obtained by fusing visible light images and infrared images, and N, M, and S are all positive integers; a first training unit, used to learn and train the original behavior detection model using the image set to obtain the target behavior detection model.

[0023] Further, the second acquisition unit includes: a first acquisition subunit, configured to acquire visible light video and infrared video corresponding to the target area; a first processing subunit, configured to perform frame extraction processing on the visible light video to obtain T visible light images, where T is greater than N and T is a positive integer; a second processing subunit, configured to perform frame extraction processing on the infrared video to obtain K infrared images, where K is less than M and K is a positive integer; a first determination subunit, configured to determine the image entropy of each visible light image and the image entropy of each infrared image respectively; a first calculation subunit, configured to calculate the ratio between the image entropy of each visible light image and the image entropy of each infrared image based on the image entropy of each visible light image and the image entropy of each infrared image, to obtain a first ratio set, wherein the first ratio set includes at least U first ratios, where U is a positive integer; and a second determination subunit, configured to determine the image set based on the U first ratios.

[0024] Further, the second determining subunit includes: a first acquisition module, configured to acquire the N visible light images from the T visible light images when there is a ratio greater than a first preset threshold among the U first ratios; a first determining module, configured to determine the M infrared images based on the K infrared images when there is a ratio less than a second preset threshold among the U first ratios, wherein the second preset threshold is less than the first preset threshold; a second determining module, configured to determine the S fused images based on the T visible light images and the K infrared images when there is a ratio greater than the second preset threshold and less than the first preset threshold among the U first ratios; and a first processing module, configured to perform summary processing on the N visible light images, the M infrared images, and the S fused images to obtain the image set.

[0025] Further, the first determining module includes: a first acquiring module, used to acquire X infrared images from the K infrared images when there is a ratio less than a second preset threshold among the U first ratios, where X is less than K and X is a positive integer; a first processing module, used to preprocess the X infrared images to obtain P preprocessed infrared images, where P is greater than X and P is a positive integer; and a first determining module, used to use the P preprocessed infrared images as the M infrared images.

[0026] Further, the first processing submodule includes: a first processing submodule, used to perform homomorphic filtering on the X infrared images to obtain X homomorphically filtered infrared images; a second processing submodule, used to perform histogram equalization on the X infrared images to obtain X histogram equalized infrared images; and a third processing submodule, used to perform summative processing on the X homomorphically filtered infrared images and the X histogram equalized infrared images to obtain the P preprocessed infrared images.

[0027] Further, the first processing submodule includes: a first secondary processing submodule, used to perform logarithmic transformation on the X infrared images to obtain X logarithmically transformed infrared images; a second secondary processing submodule, used to perform Fourier transform on the X logarithmically transformed infrared images to obtain X Fourier transformed infrared images; a third secondary processing submodule, used to perform filtering on the X Fourier transformed infrared images to obtain X filtered infrared images; and a fourth secondary processing submodule, used to perform inverse Fourier transform on the X filtered infrared images to obtain X homomorphically filtered infrared images.

[0028] Further, the second processing submodule includes: a first acquisition submodule, used to acquire the signal-to-noise ratio (SNR) of each infrared image in the X infrared images; a second acquisition submodule, used to acquire the SNR of each sub-image in each infrared image; a first calculation submodule, used to calculate the ratio between the SNR of each infrared image and the SNR of each sub-image to obtain a second ratio set, wherein the second ratio set includes at least V second ratios, where V is a positive integer; a first determination submodule, used to determine the target pixel value in each sub-image based on the second ratio set; a first removal submodule, used to remove the target pixel value from the pixel value of each sub-image to obtain a sub-image set, wherein the sub-image set includes at least Q sub-images, where Q is a positive integer; and a fifth processing submodule, used to perform histogram equalization processing on each of the Q sub-images to obtain the X histogram equalized infrared images.

[0029] Further, the second determining module includes: a second acquisition module, used to acquire Y visible light images from the T visible light images when there is a ratio among the U first ratios that is greater than the second preset threshold and less than the first preset threshold, where Y is less than T and Y is a positive integer; a third acquisition module, used to acquire R infrared images from the K infrared images, where R is less than K and R is a positive integer; and a second processing module, used to preprocess the R infrared images to obtain H preprocessed infrared images, wherein the preprocessing includes at least: homomorphic filtering and histogram equalization, where H is greater than T and Y is less than T and Y is a positive integer. Where R and H are positive integers; the second determination module is used to determine the image entropy of each visible light image in the Y visible light images and the image entropy of each infrared image in the R infrared images; the third determination module is used to determine the weight corresponding to the visible light image and the weight corresponding to the infrared image based on the image entropy of each visible light image in the Y visible light images and the image entropy of each infrared image in the R infrared images; the third processing module is used to perform fusion processing on the Y visible light images and the H preprocessed infrared images based on the weight corresponding to the visible light image and the weight corresponding to the infrared image to obtain the S fused images.

[0030] Furthermore, the device further includes: a third acquisition unit, configured to, after inputting the target image into a target behavior detection model for detection and outputting a detection result for the target image, acquire a probability value of the abnormal behavior as a target abnormal behavior if the detection result indicates that there is abnormal behavior in the target image, wherein the risk level of the target abnormal behavior is higher than a preset risk level threshold; a first judgment unit, configured to judge whether the probability value is greater than a third preset threshold; a first sending unit, configured to send a reminder message to the target object if the probability value is greater than the third preset threshold, wherein the reminder message is used to remind the target object to handle the abnormal behavior in the target image; and a first prohibition unit, configured to prohibit sending the reminder message to the target object if the probability value is not greater than the third preset threshold.

[0031] Further, the first training unit includes: a third determining subunit for determining a first loss function and a second loss function; a third processing subunit for weighting the first loss function and the second loss function to obtain a target loss function; and a first training subunit for combining the target loss function with the image set to learn and train the original behavior detection model to obtain the target behavior detection model.

[0032] To achieve the above objectives, according to another aspect of this application, a computer-readable storage medium is provided, the storage medium storing a program, wherein the program executes the abnormal behavior detection method described in any of the above claims.

[0033] To achieve the above objectives, according to another aspect of this application, an electronic device is provided, the electronic device including one or more processors and a memory, the memory being used to store one or more programs, wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the abnormal behavior detection method described in any one of the above.

[0034] This application employs the following steps: acquiring a target image to be detected, wherein the target image is the image corresponding to a target region, and the target region is the area where the financial institution is located; inputting the target image into a target behavior detection model for detection, and outputting the detection result of the target image, wherein the target behavior detection model is a model trained on the original behavior detection model based on visible light images, infrared images, and images formed by fusing visible light images and infrared images acquired in a historical process, and the detection result is used to indicate whether abnormal behavior exists in the target image, thus solving the problem of poor performance in detecting abnormal behavior in bank branches in related technologies. By acquiring an image corresponding to the area where the financial institution is located, and inputting the image corresponding to the area where the financial institution is located into a target behavior detection model (which is trained on the original behavior detection model based on visible light images, infrared images, and images formed by fusing visible light images and infrared images acquired in a historical process), and outputting the detection result of whether abnormal behavior exists in the image, the effectiveness of detecting abnormal behavior in bank branches is improved. Attached Figure Description

[0035] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0036] Figure 1 This is a flowchart of an abnormal behavior detection method provided according to an embodiment of this application;

[0037] Figure 2 This is a flowchart of the abnormal behavior detection model algorithm in the embodiments of this application;

[0038] Figure 3 This is a schematic diagram illustrating the fusion of infrared and visible light images in an embodiment of this application;

[0039] Figure 4 This is a schematic diagram of an abnormal behavior detection device provided according to an embodiment of this application;

[0040] Figure 5 This is a schematic diagram of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0041] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0042] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0043] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0044] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0045] For ease of description, the following explains some of the nouns or terms used in the embodiments of this application:

[0046] (1) Two-stage and one-stage object detection algorithms: Object detection has two branches: two-stage and one-stage. Deep learning-based two-stage object detection algorithms are based on candidate regions and include two stages: detection and recognition. The algorithm generates several bounding boxes of varying sizes and proportions based on image features such as texture, color, and detail. The detected target must exist within one of these bounding boxes. These bounding boxes are then fed into the network for object detection, hence the name "two-stage object detection algorithm." One-stage object detection algorithms (also known as one-shot object detectors) are characterized by their one-step operation and relatively fast speed. One-stage detection methods only require feeding the data into the network once to predict all bounding boxes, thus offering high speed and making them very suitable for mobile devices.

[0047] (2) R-CNN: Region-CNN is an algorithm that applies deep learning to object detection. R-CNN follows the traditional object detection approach and also uses four steps to perform object detection: extracting bounding boxes, extracting features for each box, image classification, and non-maximum suppression.

[0048] (3) Fast-RCNN: Fast Region-based Convolutional Network.

[0049] (4) Faster R-CNN: It can be seen as Fast-RCNN+RPN (Region Proposal Networks), which is an optimization based on the Fast-RCNN algorithm.

[0050] (5) YOLO series: The full name is You Only Look Once, which is a deep learning-based object detection algorithm.

[0051] (6) SSD series: The mainstream one-stage algorithms are divided into two branches: the YOLO algorithm series (YOLOv1, YOLOv2...) and the SSD algorithm series (SSD, DSSD...). YOLO (You Only Look Once) and SSD (SingleShot MultiBox Detector) are both real-time object detection algorithms.

[0052] (7) VFocalLoss: VFocalLoss is a loss function. It is often used to solve class imbalance problems in classification problems, especially when the number of positive samples is small. VFocalLoss is an improvement on Focal Loss, with better performance and stability.

[0053] (8) Focal Loss: Focal Loss is a loss function. It is widely used to solve class imbalance problems and has achieved good results in tasks such as object detection and image classification. This loss function is an improvement on the cross-entropy loss function. By introducing an adjustable parameter, it reduces the contribution of easily classified samples to the overall loss, thus paying more attention to difficult-to-classify samples.

[0054] (9) CIOU loss function: a loss function for object detection networks, derived from the IOU loss function. The IOU loss function represents the intersection-union ratio of the ground truth box to the predicted box.

[0055] (10) Backbone: The object detection network part = backbone + neck + head. The backbone part of the network is responsible for extracting features from the image. The neck is placed between the backbone and the head to better utilize the features extracted by the backbone. The head part is used for classification and localization.

[0056] The present invention will now be described in conjunction with preferred implementation steps. Figure 1 This is a flowchart of an abnormal behavior detection method provided according to an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps:

[0057] Step S101: Obtain the target image to be detected, wherein the target image is the image corresponding to the target region, and the target region is the area where the financial institution is located.

[0058] For example, the financial institution mentioned above could be a bank branch. Therefore, images (the target images mentioned above) can be obtained from the surveillance videos of the bank branch. That is, the target images mentioned above can be images generated when monitoring the area where the bank branch is located (the target area mentioned above).

[0059] Step S102: Input the target image into the target behavior detection model for detection, and output the detection result of the target image. The target behavior detection model is a model trained on the original behavior detection model based on the visible light image, infrared image and the image formed by fusing the visible light image and infrared image obtained in the historical process. The detection result is used to indicate whether there is abnormal behavior in the target image.

[0060] For example, historical surveillance videos of bank branches can be obtained first, and visible light and infrared images can be extracted from these videos and fused together to form a fused image. Then, the abnormal behavior detection model (the original behavior detection model mentioned above) can be trained using the visible light and infrared images from the historical videos, as well as the fused image, to obtain a trained abnormal behavior detection model (the target behavior detection model mentioned above). Finally, images obtained from the bank branch surveillance videos (the target images mentioned above) are input into the trained abnormal behavior detection model (the target behavior detection model mentioned above), which then detects the images and obtains the detection results indicating whether abnormal behavior exists in the images.

[0061] The above steps S101 to S102 involve acquiring an image corresponding to the area where the financial institution is located, and then inputting the image corresponding to the area where the financial institution is located into a target behavior detection model trained on the original behavior detection model based on visible light images, infrared images, and images formed by fusing visible light images and infrared images obtained in the historical process. The model then outputs the detection result of whether there is abnormal behavior in the image, thereby improving the effect of detecting abnormal behavior in bank branches.

[0062] Optionally, in the abnormal behavior detection method provided in this application embodiment, obtaining the image set includes: obtaining a visible light video and an infrared video corresponding to the target area; performing frame extraction processing on the visible light video to obtain T visible light images, where T is greater than N and T is a positive integer; performing frame extraction processing on the infrared video to obtain K infrared images, where K is less than M and K is a positive integer; determining the image entropy of each visible light image and the image entropy of each infrared image respectively; calculating the ratio between the image entropy of each visible light image and the image entropy of each infrared image based on the image entropy of each visible light image and the image entropy of each infrared image to obtain a first ratio set, wherein the first ratio set includes at least U first ratios, where U is a positive integer; and determining the image set based on the U first ratios.

[0063] For example, visible light and infrared video can be captured using a camera, with the primary event being abnormal behavior. The captured video can then be processed by frame extraction to create a dataset for an abnormal behavior detection algorithm. The image entropy of the infrared and visible light images can then be calculated, and adaptive channel fusion can be performed based on the ratio of the image entropy. The captured visible light image, infrared image, and the image fused through adaptive channel fusion are then used as the training dataset for the model (the aforementioned image set).

[0064] The above method can quickly and accurately obtain the image set used to train the model.

[0065] Optionally, in the abnormal behavior detection method provided in this application embodiment, determining the image set based on U first ratios includes: if there is a ratio among the U first ratios that is greater than a first preset threshold, obtaining N visible light images from T visible light images; if there is a ratio among the U first ratios that is less than a second preset threshold, determining M infrared images based on K infrared images, wherein the second preset threshold is less than the first preset threshold; if there is a ratio among the U first ratios that is greater than the second preset threshold and less than the first preset threshold, determining S fused images based on T visible light images and K infrared images; and summarizing the N visible light images, M infrared images, and S fused images to obtain the image set.

[0066] For example, excessively strong light or insufficient light source can cause overexposure or underexposure in the captured image. Visible light images captured in these conditions are unsuitable for target detection. Similarly, if the image is out of focus during capture, resulting in a blurry image, it is also unusable. In these situations, the histogram distribution of the visible light image will be relatively concentrated. For these cases, image entropy can be used for evaluation. Image entropy describes the average amount of information in the image source; it represents the degree of clustering in the image's grayscale distribution. Therefore, it can be used as an assessment of image blurriness and environmental conditions. The calculation formula is as follows: Among them, e K Let K represent information entropy, and N represent a pixel in the image. K The length of the image is represented by M. K The width of the image is represented by i and j, the grayscale values ​​are represented by i and j, P represents the probability of each grayscale level occurring, and C represents the probability of each grayscale level occurring. KThis represents the grayscale level. Therefore, in this embodiment, information entropy can be used to fuse infrared and visible light images. When fusing images, the information entropy of the infrared and visible light images can be calculated and normalized based on their histogram distributions. Then, the information entropy of the visible light image can be divided by the information entropy of the infrared image. If the ratio exceeds threshold A (the first preset threshold mentioned above), it indicates that the visible light image has a relatively uniform light distribution and is relatively clear. In this case, the visible light image is used for target detection, and the weight of the infrared image is set to 0, while the weight of the visible light image is set to 1. If the ratio is lower than threshold B (the second preset threshold mentioned above), it indicates that the visible light image is too blurry to be used for target detection. In this case, the infrared image is used for target detection, and the weight of the visible light image is set to 0, while the weight of the infrared image is set to 1. If the ratio of the two is between A (the first preset threshold mentioned above) and B (the second preset threshold mentioned above), it indicates that the situation may be due to insufficient light, such as dusk. In this case, the weight of the visible light image can be set to the ratio of the information entropy calculated from the visible light image to the sum of the information entropies of the two images, and the weight of the infrared image can be set to 1 minus the weight of the visible light image at this time. Then, based on the image weights for each of the above situations, the visible light image, the infrared image, and the image formed by fusing the visible light image and the infrared image are used as the aforementioned image set.

[0067] The above scheme avoids the need for additional infrared image processing when the visible light image is clear and well-lit during the day. When the visible light image is particularly blurry, the infrared image is used for recognition to avoid the problem of reduced recognition accuracy caused by information in the visible light image. When the light is insufficient, the infrared image and the visible light image are fused together as the main data source for the target detection algorithm.

[0068] Optionally, in the abnormal behavior detection method provided in the embodiments of this application, when there is a ratio less than a second preset threshold among the U first ratios, determining M infrared images based on K infrared images includes: when there is a ratio less than the second preset threshold among the U first ratios, obtaining X infrared images from the K infrared images, where X is less than K and X is a positive integer; preprocessing the X infrared images to obtain P preprocessed infrared images, where P is greater than X and P is a positive integer; and using the P preprocessed infrared images as M infrared images.

[0069] For example, if the ratio of the two is lower than threshold B (the second preset threshold mentioned above), some clear infrared images can be obtained from the infrared images captured from the original video, and these infrared images can be preprocessed using two or more preprocessing methods. Then, the preprocessed images are used as infrared images for training the model.

[0070] The above method can quickly and accurately obtain infrared images for training models.

[0071] Optionally, in the abnormal behavior detection method provided in this application embodiment, preprocessing X infrared images to obtain P preprocessed infrared images includes: performing homomorphic filtering on X infrared images to obtain X homomorphically filtered infrared images; performing histogram equalization on X infrared images to obtain X histogram equalized infrared images; and summarizing the X homomorphically filtered infrared images and the X histogram equalized infrared images to obtain P preprocessed infrared images.

[0072] For example, in this embodiment, the preprocessing method for the infrared image can employ homomorphic filtering and an adaptive contrast-limited histogram equalization algorithm. Specifically, the infrared image can be homomorphically filtered, and then preprocessed using an adaptive contrast-limited histogram equalization algorithm. The infrared image obtained through these two preprocessing methods can then be used as the preprocessed infrared image.

[0073] The above scheme, employing homomorphic filtering and adaptive contrast-limited histogram equalization algorithms, can quickly and accurately preprocess infrared images.

[0074] Optionally, in the abnormal behavior detection method provided in this application embodiment, performing homomorphic filtering on X infrared images to obtain X homomorphically filtered infrared images includes: performing logarithmic transformation on X infrared images to obtain X logarithmically transformed infrared images; performing Fourier transform on X logarithmically transformed infrared images to obtain X Fourier transformed infrared images; performing filtering on X Fourier transformed infrared images to obtain X filtered infrared images; and performing inverse Fourier transform on X filtered infrared images to obtain X homomorphically filtered infrared images.

[0075] For example, homomorphic filtering employs a linear filtering method to perform nonlinear mappings in different domains. Furthermore, an image can be viewed as the product of intensity and reflection components: f(x,y) = f i (x,y)*f rThe algorithm uses (x, y) as an example, where x and y represent the pixel values ​​of the image. Furthermore, the algorithm can first perform a logarithmic transformation on the original image: lnf(x, y) = lnf... i (x,y)+lnf r (x,y); then perform a Fourier transform on the image to obtain the corresponding frequency domain representation: DFT[lnf(x,y)]=DFT[lnf i (x,y)]+DFT[lnf r [x,y]; then filter the frequency domain, and then perform an inverse Fourier transform to obtain the exponent, to get the final result.

[0076] In summary, by performing homomorphic filtering on the acquired image, multiplicative noise can be removed while increasing the image contrast and normalized brightness.

[0077] Optionally, in the abnormal behavior detection method provided in this application embodiment, performing histogram equalization processing on X infrared images to obtain X histogram equalized infrared images includes: obtaining the signal-to-noise ratio (SNR) of each infrared image in the X infrared images; obtaining the SNR of each sub-image in each infrared image; calculating the ratio between the SNR of each infrared image and the SNR of each sub-image to obtain a second ratio set, wherein the second ratio set includes at least V second ratios, where V is a positive integer; determining the target pixel value in each sub-image based on the second ratio set; removing the target pixel value from the pixel value of each sub-image to obtain a sub-image set, wherein the sub-image set includes at least Q sub-images, where Q is a positive integer; and performing histogram equalization processing on each of the Q sub-images to obtain X histogram equalized infrared images.

[0078] For example, the adaptive contrast-limited histogram equalization algorithm is an improvement on the CLAHE (contrast-limited histogram equalization) algorithm. Because infrared images have low contrast, their grayscale distribution is usually concentrated in narrow areas. Histogram equalization can make the grayscale distribution of infrared images more uniform, thereby increasing image detail. The steps of histogram equalization are: first, calculate the grayscale probability of each pixel, and then calculate the distribution of the transformed pixels. The CLAHE algorithm is an improvement on histogram equalization. It divides the image into different regions, applies contrast-limiting to each region, and evenly distributes the portions exceeding a threshold according to a threshold. Then, it performs histogram equalization on the uniformized regions. However, this approach uses the same threshold for all regions, which can lead to insufficient noise suppression in noisy regions. This embodiment proposes a signal-to-noise ratio (SNR) based CLAHE image enhancement method: first, calculate the SNR of the entire image and the SNR of each sub-image; then, set the threshold for each sub-image as the ratio of the sub-image's SNR to the overall image's SNR; set the CLAHE threshold based on this ratio; redistribute pixels according to the threshold; and then perform histogram equalization on each region.

[0079] In summary, by combining the signal-to-noise ratio and the CLAHE method, the threshold of each region can be adaptively adjusted, and noise can be suppressed while enhancing image details.

[0080] Optionally, in the abnormal behavior detection method provided in this application embodiment, when there is a ratio among U first ratios that is greater than a second preset threshold and less than a first preset threshold, determining S fused images based on T visible light images and K infrared images includes: when there is a ratio among U first ratios that is greater than a second preset threshold and less than a first preset threshold, obtaining Y visible light images from the T visible light images, where Y is less than T and Y is a positive integer; obtaining R infrared images from the K infrared images, where R is less than K and R is a positive integer; and preprocessing the R infrared images to obtain H preprocessed images. The processed infrared images include, but are not limited to, the following preprocessing steps: homomorphic filtering and histogram equalization, where H is greater than R and H is a positive integer; determining the image entropy of each of the Y visible light images and the image entropy of each of the R infrared images; determining the weights corresponding to the visible light images and the infrared images based on the image entropy of each of the Y visible light images and the image entropy of each of the R infrared images; and fusing the Y visible light images and the H preprocessed infrared images based on the weights corresponding to the visible light images and the infrared images to obtain S fused images.

[0081] For example, if the ratio of the two is between A (the aforementioned first preset threshold) and B (the aforementioned second preset threshold), some clear visible light images can be obtained from the visible light images captured from the original video, and some clear infrared images can be obtained from the infrared images captured from the original video. Then, these infrared images can be preprocessed using two or more preprocessing methods. Furthermore, weights for the visible light images and infrared images are set separately. The set weight for the visible light images can be the information entropy calculated from the visible light images divided by the sum of the information entropies of the two images. The weight for the infrared images can be set to 1 minus the current weight of the visible light images. Then, based on the set weights for the visible light images and infrared images, the acquired clear visible light images and the preprocessed infrared images are fused to obtain the fused image.

[0082] The above method can quickly and accurately obtain the image after the fusion of visible light and infrared images.

[0083] Optionally, in the abnormal behavior detection method provided in the embodiments of this application, the target behavior detection model is obtained in the following way: acquiring an image set, wherein the image set includes at least N visible light images, M infrared images, and S fused images, the fused images being images obtained by fusing visible light images and infrared images, and N, M, and S being positive integers; using the image set to learn and train the original behavior detection model to obtain the target behavior detection model.

[0084] For example, a dataset (the aforementioned image set) can be obtained first for training the model. This dataset (the aforementioned image set) can include visible light images, infrared images, and images formed by fusing visible light and infrared images. The dataset (the aforementioned image set) is then input into the abnormal behavior recognition model for training, resulting in a trained abnormal behavior recognition model (the aforementioned target behavior detection model). Furthermore, the target detection algorithm used can be the PP-PicoDet algorithm (a target detection algorithm), which is more lightweight than mainstream target detection algorithms.

[0085] The above method can quickly and accurately obtain a trained abnormal behavior detection model.

[0086] Optionally, in the abnormal behavior detection method provided in this application embodiment, the process of using an image set to learn and train the original behavior detection model to obtain the target behavior detection model includes: determining a first loss function and a second loss function; weighting the first loss function and the second loss function to obtain a target loss function; and combining the target loss function with an image set to learn and train the original behavior detection model to obtain the target behavior detection model.

[0087] For example, the VFocalLoss and CIOU loss functions can be weighted together as the final loss function (the target loss function mentioned above). The weighted loss function (the target loss function mentioned above) and the dataset used to train the model (the image set mentioned above) are then used to train the abnormal behavior recognition model, and the trained abnormal behavior recognition model (the target behavior detection model mentioned above) is obtained.

[0088] In conclusion, by using a weighted loss function to train the model, the entire algorithm model can be more easily deployed to mobile devices.

[0089] Optionally, in the abnormal behavior detection method provided in this application embodiment, after inputting the target image into the target behavior detection model for detection and outputting the detection result of the target image, the method further includes: if the detection result indicates that there is abnormal behavior in the target image, then obtaining the probability value of the abnormal behavior as the target abnormal behavior, wherein the risk level of the target abnormal behavior is higher than a preset risk level threshold; determining whether the probability value is greater than a third preset threshold; if the probability value is greater than the third preset threshold, then sending a reminder message to the target object, wherein the reminder message is used to remind the target object to handle the abnormal behavior in the target image; if the probability value is not greater than the third preset threshold, then prohibiting the sending of the reminder message to the target object.

[0090] For example, if the identified result exceeds a certain abnormal behavior (the aforementioned target abnormal behavior) threshold (the aforementioned third preset threshold), an alarm can be triggered through the backend system, facilitating further handling of the situation by security personnel (the aforementioned target object). If the identified result does not exceed the certain abnormal behavior (the aforementioned target abnormal behavior) threshold (the aforementioned third preset threshold), no alarm will be triggered in the backend system; that is, there is no need to notify security personnel (the aforementioned target object) to handle the situation further. Furthermore, the aforementioned target abnormal behavior can be some high-risk abnormal behavior. That is, if such a high-risk abnormal behavior occurs, it will pose a significant risk to the bank branch. Therefore, when such a high-risk abnormal behavior occurs, it is necessary to notify security personnel (the aforementioned target object) to take appropriate measures.

[0091] The above-mentioned solutions can ensure the security of bank branches.

[0092] Furthermore, infrared imaging technology has advantages such as long operating range and strong anti-interference ability, and it can also perform target detection and identification even in poor ambient light conditions. However, due to the poor image quality and low signal-to-noise ratio of infrared images, they cannot be used alone for target identification.

[0093] For example, intelligent video surveillance systems can detect anomalies in monitored footage, identify different behaviors, and issue alerts and provide useful information in the fastest and most efficient way. This allows them to more effectively assist security personnel in handling crises while minimizing false alarms and missed detections. They can perform 24 / 7 long-term monitoring, continuously analyzing the monitored footage through video processing modules and abnormal behavior detection algorithms embedded in the front-end devices (video acquisition modules). Simultaneously, the front-end devices possess image processing capabilities, which, combined with the abnormal behavior detection algorithms, improve recognition accuracy. When the algorithm detects abnormal behavior, it connects to the security personnel management backend to issue an alarm. Security personnel then use the alarm video to further determine whether further measures are needed. This effectively saves manpower while reducing the probability of abnormal behavior occurring at the site.

[0094] Furthermore, since most target detection algorithms currently only use visible light images for target detection, this embodiment overcomes the problem of reduced target detection accuracy caused by the lack of light sources or uneven distribution of light sources in existing target detection algorithms. It proposes an intelligent monitoring system that can be deployed in bank branches. In this embodiment, the intelligent video monitoring system can use a visible light image and infrared image quality evaluation technology, trained by a lightweight convolutional neural network, to achieve an intelligent monitoring system that is easy to deploy and can switch algorithms according to lighting conditions, thereby achieving all-weather anomaly detection.

[0095] For example, Figure 2 This is a flowchart of the abnormal behavior detection model algorithm in this application embodiment. The main steps of the abnormal behavior detection model algorithm can be divided into: 1. Acquiring visible light abnormal behavior images and infrared abnormal behavior images; 2. Image preprocessing; 3. Adaptive channel fusion of infrared and visible light images according to evaluation criteria; 4. Inputting the fused image into the abnormal behavior detection model for training; 5. Inputting the abnormal behavior image to be identified into the trained abnormal behavior recognition model and outputting the recognition result; 6. Connecting to the security personnel's backend operation center, transmitting the result image of the detected abnormal behavior to the backend, and having the security personnel determine whether to proceed with subsequent operations. Details are as follows:

[0096] 1. Visible light and infrared video are captured through cameras. The main events captured are abnormal behaviors. The captured videos are then processed by frame extraction to form a dataset for the abnormal behavior detection algorithm.

[0097] 2. Infrared image preprocessing:

[0098] In this embodiment, the infrared image preprocessing used can include homomorphic filtering and adaptive contrast-limited histogram equalization algorithms.

[0099] (1) Homomorphic Filtering: Homomorphic filtering employs a linear filtering method with nonlinear mapping in different domains. An image can be viewed as the product of light intensity components and reflection components: f(x,y)=f i (x,y)*f r The algorithm first performs a logarithmic transformation on the original image: lnf(x,y) = lnf(x,y) i (x,y)+lnf r (x,y); then perform a Fourier transform on the image to obtain the corresponding frequency domain representation: DFT[lnf(x,y)]=DFT[lnf i (x,y)]+DFT[lnf r [x,y]; The frequency domain is filtered, and then an inverse Fourier transform is performed to obtain the exponent, yielding the final result. This algorithm can increase contrast and normalize brightness while removing multiplicative noise.

[0100] (2) Adaptive Contrast-Limited Histogram Equalization Algorithm: This algorithm is an improvement on the CLAHE (Contrast-Limited Histogram Equalization) algorithm. Because infrared images have low contrast, their grayscale distribution is usually concentrated in a narrow area. Histogram equalization can make the grayscale distribution of infrared images more uniform, thereby increasing image detail. The steps of histogram equalization are: first, calculate the grayscale probability of each pixel, and then calculate the distribution after pixel transformation. The CLAHE algorithm is an improvement on histogram equalization. It divides the image into different regions, applies contrast-limited histogram equalization to each region, and evenly distributes the portion exceeding the threshold according to a threshold. Then, it performs histogram equalization on the uniformized regions. However, this approach uses the same threshold for all regions, which can lead to insufficient noise suppression in noisy regions. This embodiment proposes a signal-to-noise ratio (SNR)-based CLAHE image enhancement method: First, the SNR of the entire image and the SNR of each sub-image are calculated. Then, the threshold for each sub-image is set as the ratio of the sub-image's SNR to the overall image's SNR. Based on this ratio, the CLAHE threshold is set, and pixels are redistributed according to the threshold. Finally, histogram equalization is performed on each region. This method, by combining SNR and CLAHE, can adaptively adjust the threshold for each region, enhancing image details while suppressing noise.

[0101] 3. Adaptive Channel Fusion of Visible and Infrared Images. When light is too strong or insufficient, the captured image may be overexposed or underexposed. In such cases, the visible light image is unfavorable for target detection. Similarly, if the image is out of focus during capture, resulting in blurriness, the image is also unusable. In these situations, the histogram distribution of the visible light image will be relatively concentrated. To address this, image entropy can be calculated. Image entropy describes the average information content of the image source and represents the degree of clustering in the image's grayscale distribution. Therefore, it can be used as an assessment of image blurriness and environmental conditions. The calculation formula is as follows: Among them, e K Let K represent information entropy, and N represent a pixel in the image. K The length of the image is represented by M. K The width of the image is represented by i and j, the grayscale values ​​are represented by i and j, P represents the probability of each grayscale level occurring, and C represents the probability of each grayscale level occurring. K This represents the grayscale level. Therefore, this embodiment can adaptively fuse infrared and visible light images by combining information entropy. First, based on the histogram distribution of the infrared and visible light images, the information entropy of the infrared and visible light images is calculated separately and normalized. Then, the information entropy of the visible light image is divided by the information entropy of the infrared image. Figure 3 This is a schematic diagram illustrating the fusion of infrared and visible light images in an embodiment of this application, as shown below. Figure 3 As shown, if the ratio of the two exceeds threshold A, it indicates that the visible light image has a relatively uniform light distribution and is clear, so the visible light image is used for target detection, with the infrared image having a weight of 0 and the visible light image having a weight of 1. If the ratio is below threshold B, it indicates that the visible light image is too blurry for target detection, so the infrared image is used, with the visible light image having a weight of 0 and the infrared image having a weight of 1. If the ratio is between A and B, it indicates that the light is insufficient, such as at dusk, and the weight of the visible light image is set to the ratio of the information entropy calculated from the visible light image to the sum of the information entropies of the two images. This avoids the situation where, when the visible light image is clear and there is sufficient light during the day, direct use of the visible light image for target detection is sufficient, eliminating the need for additional infrared image processing; when the visible light image is particularly blurry, the infrared image is used for recognition to avoid the problem of reduced recognition accuracy caused by information in the visible light image; and when there is insufficient light, the infrared and visible light images are fused as the main data source for the target detection algorithm. Furthermore, Figure 3 Enhancement algorithm 1 in the text can be a homomorphic filtering algorithm. Figure 3 The enhancement algorithm 2 in the text can be an adaptive contrast-limited histogram equalization algorithm.

[0102] 4. The fused image is input into the abnormal behavior recognition model for training. The target detection algorithm used is the PP-PicoDet algorithm (a target detection algorithm). This algorithm is more lightweight than mainstream target detection algorithms. It uses VFocalLoss and CIOU loss functions as weighted loss functions as the final loss function, making the whole algorithm easier to deploy on mobile devices.

[0103] 5. The trained network model is used as the final abnormal behavior detection model. Inputting an image of the abnormal behavior to be detected will output the recognition result.

[0104] 6. If the identified results exceed a certain abnormal behavior threshold, an alarm will be triggered through the backend system, facilitating further handling of the situation by security personnel.

[0105] Moreover, the method provided by the embodiments of this application can have the following beneficial effects:

[0106] (1) In this embodiment, an adaptive visible light image and infrared image adaptive channel-level fusion method is proposed. The method combines visible light image and infrared image for abnormal behavior recognition, which can effectively solve the problem of reduced abnormal behavior recognition rate when using visible light image for target detection when the environment lacks light source or the light is too bright. It can save manpower and monitor abnormal behavior around the clock.

[0107] (2) In this embodiment, an adaptive contrast-limited histogram equalization technique combining signal-to-noise ratio and CLAHE is proposed, and a homomorphic filtering algorithm is used to preprocess the infrared image, which effectively improves the quality of the infrared image and solves the problem of the scarcity of infrared images.

[0108] (3) In this embodiment, the PP-PicoDet algorithm (a target detection algorithm) is used as the abnormal behavior recognition model, and the VFocalLoss and CIOU loss functions are weighted as the final loss function, making the whole algorithm easier to deploy.

[0109] In addition, other network models can be used to replace the target detection network in this embodiment. This embodiment only uses a relatively accurate network model from the current open source as the target detection model. You can find a backbone or loss function that can further improve the accuracy to replace it.

[0110] In summary, the abnormal behavior detection method provided in this application acquires a target image to be detected, wherein the target image is the image corresponding to a target region, and the target region is the area where the financial institution is located; the target image is input into a target behavior detection model for detection, and the detection result of the target image is output. The target behavior detection model is a model trained on an original behavior detection model based on visible light images, infrared images, and an image formed by fusing visible light and infrared images acquired in a historical process. The detection result is used to indicate whether abnormal behavior exists in the target image, thus solving the problem of poor performance in detecting abnormal behavior in bank branches in related technologies. By acquiring an image corresponding to the area where the financial institution is located, and inputting this image into a target behavior detection model trained on an original behavior detection model based on visible light images, infrared images, and an image formed by fusing visible light and infrared images acquired in a historical process, the method outputs a detection result indicating whether abnormal behavior exists in the image, thereby improving the effectiveness of detecting abnormal behavior in bank branches.

[0111] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0112] This application also provides an abnormal behavior detection device. It should be noted that the abnormal behavior detection device of this application can be used to execute the abnormal behavior detection method provided in this application. The abnormal behavior detection device provided in this application is described below.

[0113] Figure 4 This is a schematic diagram of an abnormal behavior detection device according to an embodiment of this application. Figure 4 As shown, the device includes a first acquisition unit 401 and a first detection unit 402.

[0114] Specifically, the first acquisition unit 401 is used to acquire the target image to be detected, wherein the target image is the image corresponding to the target region, and the target region is the region where the financial institution is located;

[0115] The first detection unit 402 is used to input the target image into the target behavior detection model for detection and output the detection result of the target image. The target behavior detection model is a model trained on the original behavior detection model based on the visible light image, infrared image and the image formed by fusing the visible light image and infrared image obtained in the historical process. The detection result is used to indicate whether there is abnormal behavior in the target image.

[0116] In summary, the abnormal behavior detection device provided in this application acquires a target image to be detected through a first acquisition unit 401. The target image is an image corresponding to a target region, which is the area where the financial institution is located. The first detection unit 402 inputs the target image into a target behavior detection model for detection and outputs the detection result for the target image. The target behavior detection model is a model trained on an original behavior detection model using visible light images, infrared images acquired in a historical process, and an image formed by fusing visible light and infrared images. The detection result indicates whether abnormal behavior exists in the target image, solving the problem of poor detection of abnormal behavior in bank branches in related technologies. By acquiring an image corresponding to the area where the financial institution is located and inputting this image into a target behavior detection model trained on an original behavior detection model using visible light images, infrared images acquired in a historical process, and an image formed by fusing visible light and infrared images, the device outputs a detection result indicating whether abnormal behavior exists in the image, thereby improving the effectiveness of detecting abnormal behavior in bank branches.

[0117] Optionally, in the abnormal behavior detection device provided in this application embodiment, the target behavior detection model is obtained in the following manner: a second acquisition unit is used to acquire an image set, wherein the image set includes at least N visible light images, M infrared images, and S fused images, the fused images being images obtained by fusing visible light images and infrared images, and N, M, and S are all positive integers; a first training unit is used to learn and train the original behavior detection model using the image set to obtain the target behavior detection model.

[0118] Optionally, in the abnormal behavior detection device provided in this application embodiment, the second acquisition unit includes: a first acquisition subunit, used to acquire visible light video and infrared video corresponding to the target area; a first processing subunit, used to perform frame extraction processing on the visible light video to obtain T visible light images, where T is greater than N and T is a positive integer; a second processing subunit, used to perform frame extraction processing on the infrared video to obtain K infrared images, where K is less than M and K is a positive integer; a first determination subunit, used to determine the image entropy of each visible light image and the image entropy of each infrared image respectively; a first calculation subunit, used to calculate the ratio between the image entropy of each visible light image and the image entropy of each infrared image based on the image entropy of each visible light image and the image entropy of each infrared image, to obtain a first ratio set, wherein the first ratio set includes at least U first ratios, where U is a positive integer; and a second determination subunit, used to determine an image set based on the U first ratios.

[0119] Optionally, in the abnormal behavior detection device provided in this application embodiment, the second determining subunit includes: a first acquisition module, configured to acquire N visible light images from T visible light images when there is a ratio greater than a first preset threshold among U first ratios; a first determining module, configured to determine M infrared images based on K infrared images when there is a ratio less than a second preset threshold among U first ratios, wherein the second preset threshold is less than the first preset threshold; a second determining module, configured to determine S fused images based on T visible light images and K infrared images when there is a ratio greater than the second preset threshold and less than the first preset threshold among U first ratios; and a first processing module, configured to perform summary processing on the N visible light images, M infrared images, and S fused images to obtain an image set.

[0120] Optionally, in the abnormal behavior detection device provided in this application embodiment, the first determining module includes: a first acquisition sub-module, used to acquire X infrared images from K infrared images when there is a ratio less than a second preset threshold among U first ratios, where X is less than K and X is a positive integer; a first processing sub-module, used to preprocess the X infrared images to obtain P preprocessed infrared images, where P is greater than X and P is a positive integer; and a first determining sub-module, used to take the P preprocessed infrared images as M infrared images.

[0121] Optionally, in the abnormal behavior detection device provided in this application embodiment, the first processing sub-module includes: a first processing sub-module, used to perform homomorphic filtering on X infrared images to obtain X homomorphically filtered infrared images; a second processing sub-module, used to perform histogram equalization on X infrared images to obtain X histogram equalized infrared images; and a third processing sub-module, used to perform summative processing on the X homomorphically filtered infrared images and the X histogram equalized infrared images to obtain P preprocessed infrared images.

[0122] Optionally, in the abnormal behavior detection device provided in this application embodiment, the first processing submodule includes: a first secondary processing submodule, used to perform logarithmic transformation processing on X infrared images to obtain X infrared images after logarithmic transformation processing; a second secondary processing submodule, used to perform Fourier transform processing on the X infrared images after logarithmic transformation processing to obtain X infrared images after Fourier transform processing; a third secondary processing submodule, used to perform filtering processing on the X infrared images after Fourier transform processing to obtain X infrared images after filtering processing; and a fourth secondary processing submodule, used to perform inverse Fourier transform processing on the X infrared images after filtering processing to obtain X infrared images after homomorphic filtering processing.

[0123] Optionally, in the abnormal behavior detection device provided in this application embodiment, the second processing submodule includes: a first acquisition submodule, used to acquire the signal-to-noise ratio (SNR) of each infrared image in X infrared images; a second acquisition submodule, used to acquire the SNR of each sub-image in each infrared image; a first calculation submodule, used to calculate the ratio between the SNR of each infrared image and the SNR of each sub-image to obtain a second ratio set, wherein the second ratio set includes at least V second ratios, where V is a positive integer; a first determination submodule, used to determine the target pixel value in each sub-image based on the second ratio set; a first removal submodule, used to remove the target pixel value from the pixel value of each sub-image to obtain a sub-image set, wherein the sub-image set includes at least Q sub-images, where Q is a positive integer; and a fifth processing submodule, used to perform histogram equalization processing on each of the Q sub-images to obtain X infrared images after histogram equalization processing.

[0124] Optionally, in the abnormal behavior detection device provided in this application embodiment, the second determining module includes: a second acquisition sub-module, used to acquire Y visible light images from T visible light images when there is a ratio among U first ratios that is greater than a second preset threshold and less than a first preset threshold, wherein Y is less than T and Y is a positive integer; a third acquisition sub-module, used to acquire R infrared images from K infrared images, wherein R is less than K and R is a positive integer; and a second processing sub-module, used to preprocess the R infrared images to obtain H preprocessed infrared images, wherein the preprocessing includes at least: homomorphic filtering and direct current processing. The first module performs graph equalization, where H is greater than R and H is a positive integer. The second module determines the image entropy of each visible light image in the Y visible light images and the image entropy of each infrared image in the R infrared images. The third module determines the weights corresponding to the visible light images and the infrared images based on the image entropy of each visible light image in the Y visible light images and the image entropy of each infrared image in the R infrared images. The third module performs fusion processing on the Y visible light images and the H preprocessed infrared images based on the weights corresponding to the visible light images and the infrared images, resulting in S fused images.

[0125] Optionally, in the abnormal behavior detection device provided in this application embodiment, the device further includes: a third acquisition unit, configured to, after inputting the target image into the target behavior detection model for detection and outputting the detection result of the target image, acquire a probability value of the abnormal behavior as the target abnormal behavior if the detection result indicates that there is abnormal behavior in the target image, wherein the risk level of the target abnormal behavior is higher than a preset risk level threshold; a first judgment unit, configured to judge whether the probability value is greater than a third preset threshold; a first sending unit, configured to send a reminder message to the target object if the probability value is greater than the third preset threshold, wherein the reminder message is used to remind the target object to handle the abnormal behavior in the target image; and a first prohibition unit, configured to prohibit sending the reminder message to the target object if the probability value is not greater than the third preset threshold.

[0126] Optionally, in the abnormal behavior detection device provided in the embodiments of this application, the first training unit includes: a third determining subunit, used to determine a first loss function and a second loss function; a third processing subunit, used to perform weighted processing on the first loss function and the second loss function to obtain a target loss function; and a first training subunit, used to combine the target loss function and use an image set to learn and train the original behavior detection model to obtain a target behavior detection model.

[0127] The abnormal behavior detection device includes a processor and a memory. The first acquisition unit 401 and the first detection unit 402 are stored in the memory as program units. The processor executes the program units stored in the memory to achieve the corresponding functions.

[0128] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and adjusting kernel parameters can improve the effectiveness of detecting abnormal behavior in bank branches.

[0129] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0130] This invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the method for detecting the abnormal behavior.

[0131] This invention provides a processor for running a program, wherein the program executes the method for detecting the abnormal behavior during runtime.

[0132] like Figure 5As shown, this embodiment of the invention provides an electronic device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs the following steps: acquiring a target image to be detected, wherein the target image is an image corresponding to a target region, and the target region is the area where the financial institution is located; inputting the target image into a target behavior detection model for detection, and outputting a detection result for the target image, wherein the target behavior detection model is a model trained on an original behavior detection model based on visible light images, infrared images, and an image formed by fusing the visible light images and the infrared images acquired in a historical process, and the detection result is used to indicate whether there is abnormal behavior in the target image.

[0133] When the processor executes the program, it also performs the following steps: the target behavior detection model is obtained by: acquiring an image set, wherein the image set includes at least N visible light images, M infrared images, and S fused images, wherein the fused images are images obtained by fusing visible light images and infrared images, and N, M, and S are all positive integers; using the image set to learn and train the original behavior detection model to obtain the target behavior detection model.

[0134] The processor, when executing the program, also performs the following steps: acquiring the image set includes: acquiring visible light video and infrared video corresponding to the target area; performing frame extraction processing on the visible light video to obtain T visible light images, where T is greater than N and T is a positive integer; performing frame extraction processing on the infrared video to obtain K infrared images, where K is less than M and K is a positive integer; determining the image entropy of each visible light image and the image entropy of each infrared image respectively; calculating the ratio between the image entropy of each visible light image and the image entropy of each infrared image based on the image entropy of each visible light image and the image entropy of each infrared image to obtain a first ratio set, wherein the first ratio set includes at least U first ratios, where U is a positive integer; and determining the image set based on the U first ratios.

[0135] When the processor executes the program, it also performs the following steps: determining the image set based on the U first ratios includes: if there is a ratio among the U first ratios that is greater than a first preset threshold, obtaining the N visible light images from the T visible light images; if there is a ratio among the U first ratios that is less than a second preset threshold, determining the M infrared images based on the K infrared images, wherein the second preset threshold is less than the first preset threshold; if there is a ratio among the U first ratios that is greater than the second preset threshold and less than the first preset threshold, determining the S fused images based on the T visible light images and the K infrared images; and summarizing the N visible light images, the M infrared images, and the S fused images to obtain the image set.

[0136] When the processor executes the program, it also performs the following steps: If one of the U first ratios is less than a second preset threshold, determining the M infrared images based on the K infrared images includes: if one of the U first ratios is less than the second preset threshold, obtaining X infrared images from the K infrared images, where X is less than K and X is a positive integer; preprocessing the X infrared images to obtain P preprocessed infrared images, where P is greater than X and P is a positive integer; and using the P preprocessed infrared images as the M infrared images.

[0137] When the processor executes the program, it also performs the following steps: preprocessing the X infrared images to obtain P preprocessed infrared images, including: performing homomorphic filtering on the X infrared images to obtain X homomorphically filtered infrared images; performing histogram equalization on the X infrared images to obtain X histogram equalized infrared images; and summarizing the X homomorphically filtered infrared images and the X histogram equalized infrared images to obtain the P preprocessed infrared images.

[0138] When the processor executes the program, it also performs the following steps: performing homomorphic filtering on the X infrared images to obtain X homomorphically filtered infrared images, including: performing logarithmic transformation on the X infrared images to obtain X logarithmically transformed infrared images; performing Fourier transform on the X logarithmically transformed infrared images to obtain X Fourier transformed infrared images; performing filtering on the X Fourier transformed infrared images to obtain X filtered infrared images; and performing inverse Fourier transform on the X filtered infrared images to obtain the X homomorphically filtered infrared images.

[0139] The processor, when executing the program, also performs the following steps: performing histogram equalization on the X infrared images to obtain X histogram-equalized infrared images, including: obtaining the signal-to-noise ratio (SNR) of each infrared image in the X infrared images; obtaining the SNR of each sub-image in each infrared image; calculating the ratio between the SNR of each infrared image and the SNR of each sub-image to obtain a second ratio set, wherein the second ratio set includes at least V second ratios, where V is a positive integer; determining the target pixel value in each sub-image based on the second ratio set; removing the target pixel value from the pixel value of each sub-image to obtain a sub-image set, wherein the sub-image set includes at least Q sub-images, where Q is a positive integer; and performing histogram equalization on each of the Q sub-images to obtain the X histogram-equalized infrared images.

[0140] When the processor executes the program, it also performs the following steps: If, among the U first ratios, there exists a ratio greater than the second preset threshold and less than the first preset threshold, determining the S fused images based on the T visible light images and the K infrared images includes: if, among the U first ratios, there exists a ratio greater than the second preset threshold and less than the first preset threshold, obtaining Y visible light images from the T visible light images, where Y is less than T and Y is a positive integer; obtaining R infrared images from the K infrared images, where R is less than K and R is a positive integer; and preprocessing the R infrared images to obtain H preprocessed images. The preprocessing of the infrared images includes at least: homomorphic filtering and histogram equalization, where H is greater than R and H is a positive integer; determining the image entropy of each of the Y visible light images and the image entropy of each of the R infrared images; determining the weights corresponding to the visible light images and the infrared images based on the image entropy of each of the Y visible light images and the image entropy of each of the R infrared images; and fusing the Y visible light images and the H preprocessed infrared images based on the weights corresponding to the visible light images and the infrared images to obtain the S fused images.

[0141] The processor, when executing the program, further implements the following steps: after inputting the target image into the target behavior detection model for detection and outputting the detection result of the target image, the method further includes: if the detection result indicates that there is abnormal behavior in the target image, then obtaining a probability value of the abnormal behavior being a target abnormal behavior, wherein the risk level of the target abnormal behavior is higher than a preset risk level threshold; determining whether the probability value is greater than a third preset threshold; if the probability value is greater than the third preset threshold, then sending a reminder message to the target object, wherein the reminder message is used to remind the target object to handle the abnormal behavior in the target image; if the probability value is not greater than the third preset threshold, then prohibiting the sending of the reminder message to the target object.

[0142] The processor, when executing the program, also performs the following steps: training the original behavior detection model using the image set to obtain the target behavior detection model, including: determining a first loss function and a second loss function; weighting the first loss function and the second loss function to obtain a target loss function; and combining the target loss function with the image set to train the original behavior detection model to obtain the target behavior detection model.

[0143] The devices mentioned in this article can be servers, PCs, tablets, mobile phones, etc.

[0144] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having the following method steps: acquiring a target image to be detected, wherein the target image is an image corresponding to a target region, and the target region is the region where the financial institution is located; inputting the target image into a target behavior detection model for detection, and outputting a detection result for the target image, wherein the target behavior detection model is a model trained on an original behavior detection model based on visible light images, infrared images, and an image formed by fusing the visible light images and the infrared images acquired in a historical process, and the detection result is used to indicate whether there is abnormal behavior in the target image.

[0145] When executed on a data processing device, it is also suitable to execute an initialization program with the following method steps: the target behavior detection model is obtained by: acquiring an image set, wherein the image set includes at least N visible light images, M infrared images, and S fused images, wherein the fused images are images obtained by fusing visible light images and infrared images, and N, M, and S are all positive integers; using the image set to learn and train the original behavior detection model to obtain the target behavior detection model.

[0146] When executed on a data processing device, it is also suitable to execute an initialization program with the following method steps: acquiring an image set includes: acquiring a visible light video corresponding to the target area and an infrared video corresponding to the target area; performing frame extraction processing on the visible light video to obtain T visible light images, where T is greater than N and T is a positive integer; performing frame extraction processing on the infrared video to obtain K infrared images, where K is less than M and K is a positive integer; determining the image entropy of each visible light image and the image entropy of each infrared image respectively; calculating the ratio between the image entropy of each visible light image and the image entropy of each infrared image based on the image entropy of each visible light image and the image entropy of each infrared image to obtain a first ratio set, wherein the first ratio set includes at least U first ratios, where U is a positive integer; and determining the image set based on the U first ratios.

[0147] When executed on a data processing device, it is also suitable to execute an initialization program with the following steps: determining the image set based on the U first ratios includes: if there is a ratio among the U first ratios that is greater than a first preset threshold, obtaining the N visible light images from the T visible light images; if there is a ratio among the U first ratios that is less than a second preset threshold, determining the M infrared images based on the K infrared images, wherein the second preset threshold is less than the first preset threshold; if there is a ratio among the U first ratios that is greater than the second preset threshold and less than the first preset threshold, determining the S fused images based on the T visible light images and the K infrared images; and summarizing the N visible light images, the M infrared images, and the S fused images to obtain the image set.

[0148] When executed on a data processing device, it is also suitable to execute an initialization program with the following steps: If one of the U first ratios is less than a second preset threshold, determining the M infrared images based on the K infrared images includes: if one of the U first ratios is less than the second preset threshold, obtaining X infrared images from the K infrared images, where X is less than K and X is a positive integer; preprocessing the X infrared images to obtain P preprocessed infrared images, where P is greater than X and P is a positive integer; and using the P preprocessed infrared images as the M infrared images.

[0149] When executed on a data processing device, it is also suitable to execute an initialization program with the following steps: preprocessing the X infrared images to obtain P preprocessed infrared images, including: performing homomorphic filtering on the X infrared images to obtain X homomorphically filtered infrared images; performing histogram equalization on the X infrared images to obtain X histogram equalized infrared images; and performing a summation process on the X homomorphically filtered infrared images and the X histogram equalized infrared images to obtain the P preprocessed infrared images.

[0150] When executed on a data processing device, it is also suitable to execute an initialization program with the following steps: performing homomorphic filtering on the X infrared images to obtain X homomorphically filtered infrared images, including: performing logarithmic transformation on the X infrared images to obtain X logarithmically transformed infrared images; performing Fourier transform on the X logarithmically transformed infrared images to obtain X Fourier transformed infrared images; performing filtering on the X Fourier transformed infrared images to obtain X filtered infrared images; and performing inverse Fourier transform on the X filtered infrared images to obtain the X homomorphically filtered infrared images.

[0151] When executed on a data processing device, it is also suitable to execute an initialization program with the following steps: performing histogram equalization on the X infrared images to obtain X histogram equalized infrared images, including: obtaining the signal-to-noise ratio (SNR) of each of the X infrared images; obtaining the SNR of each sub-image in each infrared image; calculating the ratio between the SNR of each infrared image and the SNR of each sub-image to obtain a second ratio set, wherein the second ratio set includes at least V second ratios, where V is a positive integer; determining the target pixel value in each sub-image based on the second ratio set; removing the target pixel value from the pixel value of each sub-image to obtain a sub-image set, wherein the sub-image set includes at least Q sub-images, where Q is a positive integer; and performing histogram equalization on each of the Q sub-images to obtain the X histogram equalized infrared images.

[0152] When executed on a data processing device, it is also suitable to execute an initialization program with the following steps: If, among the U first ratios, there exists a ratio greater than the second preset threshold and less than the first preset threshold, determining the S fused images based on the T visible light images and the K infrared images includes: if, among the U first ratios, there exists a ratio greater than the second preset threshold and less than the first preset threshold, obtaining Y visible light images from the T visible light images, where Y is less than T and Y is a positive integer; obtaining R infrared images from the K infrared images, where R is less than K and R is a positive integer; and preprocessing the R infrared images. The process involves obtaining H preprocessed infrared images, wherein the preprocessing includes at least homomorphic filtering and histogram equalization, where H is greater than R and H is a positive integer; determining the image entropy of each visible light image in the Y visible light images and the image entropy of each infrared image in the R infrared images; determining the weights corresponding to the visible light images and the infrared images based on the image entropy of each visible light image in the Y visible light images and the image entropy of each infrared image in the R infrared images; and fusing the Y visible light images and the H preprocessed infrared images based on the weights corresponding to the visible light images and the infrared images to obtain S fused images.

[0153] When executed on a data processing device, the method is also adapted to execute an initialization procedure with the following steps: after inputting the target image into a target behavior detection model for detection and outputting the detection result of the target image, the method further includes: if the detection result indicates that there is abnormal behavior in the target image, then obtaining a probability value of the abnormal behavior as a target abnormal behavior, wherein the risk level of the target abnormal behavior is higher than a preset risk level threshold; determining whether the probability value is greater than a third preset threshold; if the probability value is greater than the third preset threshold, then sending a reminder message to the target object, wherein the reminder message is used to remind the target object to handle the abnormal behavior in the target image; if the probability value is not greater than the third preset threshold, then prohibiting the sending of the reminder message to the target object.

[0154] When executed on a data processing device, it is also suitable to execute an initialization program with the following steps: training the original behavior detection model using the image set to obtain the target behavior detection model, including: determining a first loss function and a second loss function; weighting the first loss function and the second loss function to obtain a target loss function; and combining the target loss function with the image set to train the original behavior detection model to obtain the target behavior detection model.

[0155] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0156] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0157] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0158] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0159] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0160] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0161] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0162] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0163] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0164] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method of detecting abnormal behavior, characterized by, include: Acquire the target image to be detected, wherein the target image is the image corresponding to the target region, and the target region is the area where the financial institution is located; The target image is input into the target behavior detection model for detection, and the detection result of the target image is output. The target behavior detection model is a model trained on the original behavior detection model based on visible light images, infrared images, and images formed by fusing the visible light images and the infrared images obtained in the historical process. The detection result is used to indicate whether there is abnormal behavior in the target image. The target behavior detection model is obtained by: acquiring an image set, wherein the image set includes at least N visible light images, M infrared images, and S fused images, wherein the fused images are images obtained by fusing visible light images and infrared images, and N, M, and S are all positive integers; and using the image set to learn and train the original behavior detection model to obtain the target behavior detection model. Acquiring the image set includes: acquiring a visible light video and an infrared video corresponding to the target region; performing frame extraction on the visible light video to obtain T visible light images, where T is greater than N and T is a positive integer; performing frame extraction on the infrared video to obtain K infrared images, where K is less than M and K is a positive integer; determining the image entropy of each visible light image and the image entropy of each infrared image; calculating the ratio between the image entropy of each visible light image and the image entropy of each infrared image based on the image entropy of each visible light image and the image entropy of each infrared image to obtain a first ratio set, wherein the first ratio set includes at least U first ratios, where U is a positive integer; and determining the image set based on the U first ratios. The determination of the image set based on the U first ratios includes: if there is a ratio among the U first ratios greater than a first preset threshold, obtaining the N visible light images from the T visible light images; if there is a ratio among the U first ratios less than a second preset threshold, determining the M infrared images based on the K infrared images, wherein the second preset threshold is less than the first preset threshold; if there is a ratio among the U first ratios greater than the second preset threshold and less than the first preset threshold, determining the S fused images based on the T visible light images and the K infrared images; and summarizing the N visible light images, the M infrared images, and the S fused images to obtain the image set. When the first ratio is greater than the first preset threshold, the visible light image is used for target detection; when the first ratio is less than the second preset threshold, the infrared image is used for target detection; when the first ratio is greater than the second preset threshold and less than the first preset threshold, the fused image is used for target detection.

2. The method according to claim 1, characterized in that, If any of the U first ratios is less than a second preset threshold, determining the M infrared images based on the K infrared images includes: If any of the U first ratios is less than a second preset threshold, X infrared images are obtained from the K infrared images, where X is less than K and X is a positive integer; The X infrared images are preprocessed to obtain P preprocessed infrared images, where P is greater than X and P is a positive integer; The P preprocessed infrared images are used as the M infrared images.

3. The method according to claim 2, characterized in that, The X infrared images are preprocessed to obtain P preprocessed infrared images, including: The X infrared images are subjected to homomorphic filtering to obtain X homomorphically filtered infrared images; Histogram equalization is performed on the X infrared images to obtain X infrared images after histogram equalization. The X infrared images after homomorphic filtering and the X infrared images after histogram equalization are combined to obtain the P preprocessed infrared images.

4. The method according to claim 3, characterized in that, The X infrared images are subjected to homomorphic filtering to obtain X homomorphically filtered infrared images, including: The X infrared images are subjected to logarithmic transformation to obtain X logarithmically transformed infrared images; Perform Fourier transform processing on the X logarithmically transformed infrared images to obtain X Fourier transformed infrared images. The X infrared images after Fourier transform processing are filtered to obtain X filtered infrared images. The X filtered infrared images are subjected to inverse Fourier transform to obtain the X homomorphically filtered infrared images.

5. The method according to claim 3, characterized in that, Histogram equalization is performed on the X infrared images to obtain X histogram equalized infrared images, including: Obtain the signal-to-noise ratio of each of the X infrared images; Obtain the signal-to-noise ratio of each sub-image in each infrared image; Calculate the ratio between the signal-to-noise ratio of each infrared image and the signal-to-noise ratio of each sub-image to obtain a second set of ratios, wherein the second set of ratios includes at least V second ratios, where V is a positive integer; Based on the second set of ratios, determine the target pixel value in each sub-image; The target pixel value is removed from the pixel value of each sub-image to obtain a set of sub-images, wherein the set of sub-images includes at least Q sub-images, where Q is a positive integer; Histogram equalization is performed on each of the Q sub-images to obtain the X infrared images after histogram equalization.

6. The method according to claim 1, characterized in that, If, among the U first ratios, there exists a ratio greater than the second preset threshold and less than the first preset threshold, determining the S fused images based on the T visible light images and the K infrared images includes: If among the U first ratios there exists a ratio greater than the second preset threshold and less than the first preset threshold, Y visible light images are obtained from the T visible light images, where Y is less than T and Y is a positive integer; R infrared images are obtained from the K infrared images, where R is less than K and R is a positive integer; The R infrared images are preprocessed to obtain H preprocessed infrared images, wherein the preprocessing includes at least: homomorphic filtering and histogram equalization, H is greater than R, and H is a positive integer; Determine the image entropy of each visible light image in the Y visible light images and the image entropy of each infrared image in the R infrared images; Based on the image entropy of each visible light image in the Y visible light images and the image entropy of each infrared image in the R infrared images, the weights corresponding to the visible light images and the infrared images are determined. Based on the weights corresponding to the visible light images and the infrared images, the Y visible light images and the H preprocessed infrared images are fused to obtain the S fused images.

7. The method according to claim 1, characterized in that, After inputting the target image into the target behavior detection model for detection and outputting the detection result of the target image, the method further includes: If the detection result indicates that there is abnormal behavior in the target image, then the probability value of the abnormal behavior being the target abnormal behavior is obtained, wherein the risk level of the target abnormal behavior is higher than a preset risk level threshold. Determine whether the probability value is greater than a third preset threshold; If the probability value is greater than the third preset threshold, a reminder message is sent to the target object, wherein the reminder message is used to remind the target object to handle abnormal behavior in the target image; If the probability value is not greater than the third preset threshold, then sending the reminder information to the target object is prohibited.

8. The method according to claim 1, characterized in that, The target behavior detection model is obtained by training the original behavior detection model using the image set, including: Determine the first loss function and the second loss function; The first loss function and the second loss function are weighted to obtain the target loss function; By combining the target loss function and using the image set to learn and train the original behavior detection model, the target behavior detection model is obtained.

9. A device for detecting abnormal behavior, characterized in that, include: The first acquisition unit is used to acquire the target image to be detected, wherein the target image is the image corresponding to the target region, and the target region is the region where the financial institution is located; The first detection unit is used to input the target image into the target behavior detection model for detection and output the detection result of the target image. The target behavior detection model is a model trained on the original behavior detection model based on visible light images, infrared images, and images formed by fusing the visible light images and the infrared images obtained in the historical process. The detection result is used to indicate whether there is abnormal behavior in the target image. The target behavior detection model is obtained through the following methods: a second acquisition unit, used to acquire an image set, wherein the image set includes at least N visible light images, M infrared images, and S fused images, wherein the fused images are images obtained by fusing visible light images and infrared images, and N, M, and S are all positive integers; and a first training unit, used to train the original behavior detection model using the image set to obtain the target behavior detection model. The second acquisition unit includes: a first acquisition subunit, configured to acquire visible light video and infrared video corresponding to the target area; a first processing subunit, configured to perform frame extraction processing on the visible light video to obtain T visible light images, where T is greater than N and T is a positive integer; a second processing subunit, configured to perform frame extraction processing on the infrared video to obtain K infrared images, where K is less than M and K is a positive integer; a first determination subunit, configured to determine the image entropy of each visible light image and the image entropy of each infrared image respectively; a first calculation subunit, configured to calculate the ratio between the image entropy of each visible light image and the image entropy of each infrared image based on the image entropy of each visible light image and the image entropy of each infrared image, to obtain a first ratio set, wherein the first ratio set includes at least U first ratios, where U is a positive integer; and a second determination subunit, configured to determine the image set based on the U first ratios. The second determining subunit includes: a first acquisition module, configured to acquire the N visible light images from the T visible light images when there is a ratio greater than a first preset threshold among the U first ratios; a first determining module, configured to determine the M infrared images based on the K infrared images when there is a ratio less than a second preset threshold among the U first ratios, wherein the second preset threshold is less than the first preset threshold; a second determining module, configured to determine the S fused images based on the T visible light images and the K infrared images when there is a ratio greater than the second preset threshold and less than the first preset threshold among the U first ratios; and a first processing module, configured to perform summary processing on the N visible light images, the M infrared images, and the S fused images to obtain the image set. When the first ratio is greater than the first preset threshold, the visible light image is used for target detection; when the first ratio is less than the second preset threshold, the infrared image is used for target detection; when the first ratio is greater than the second preset threshold and less than the first preset threshold, the fused image is used for target detection.

10. A computer-readable storage medium, characterized in that, The storage medium stores a program, wherein the program executes the abnormal behavior detection method according to any one of claims 1 to 8.

11. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the abnormal behavior detection method according to any one of claims 1 to 8.

Images