Method for abnormal behavior analysis and information management based on blockchain

Abstract

The present invention is a blockchain-based abnormal behavior analysis and information management method performed by a computing device, comprising: a step of acquiring real-time video data from a video collection device; a step of detecting an object from the acquired video data; a step of identifying abnormal behavior by analyzing the behavioral pattern of the detected object, and generating a security event when the identified abnormal behavior satisfies at least one of a predetermined condition of suspected theft or a condition of non-payment within a time period; and a step of storing object information related to the abnormal behavior in a blockchain.

Description

Technology Field

[0001] The present invention relates to artificial intelligence-based video analysis technology, and more specifically, to blockchain-based storage technology for utilizing CCTV footage to detect abnormal behavior of people or objects in real time, determine whether theft has occurred, and protect personal information and share crime information between stores. Background Technology

[0002] Conventional retail stores and unmanned stores are vulnerable to theft and robbery, and most security systems rely on simple sensor-based detection or post-hoc review of recorded footage. As a result, real-time theft detection and preemptive response are difficult, and the detection rate is very low, particularly for complex abnormal behaviors such as financial theft by internal employees or self-checkout fraud.

[0003] Furthermore, because footage collected from CCTVs contains personally identifiable information, there are significant legal restrictions on the storage and use of video data, posing a problem that makes it difficult to share actual crime information.

[0004] Furthermore, current AI-based security systems are limited to fragmentary behavioral analysis or video storage, making it difficult to establish a cooperative defense system among distributed stores. Accordingly, a new approach is required that integrates real-time analysis, behavior-based detection, personal information processing considering legal protection, and blockchain-based information sharing.

[0005] The technology forming the background of the present invention is disclosed in Korean Published Patent Application No. 10-2019-0106862. The problem to be solved

[0006] The present invention aims to solve the problems of the aforementioned conventional technology by providing a blockchain-based abnormal behavior analysis and information management method that detects abnormal behavior, such as theft, through CCTV-based real-time video analysis and stores and shares this information on a blockchain basis, thereby enabling the establishment of a theft response system based not only on the store level but also on the region level.

[0007] However, the technical problems that the embodiments of the present invention aim to solve are not limited to the technical problems described above, and other technical problems may exist. means of solving the problem

[0008] As a technical means for achieving the above-mentioned technical problem, a blockchain-based abnormal behavior analysis and information management method is disclosed, which is performed by a computing device according to an embodiment of the present invention. The method may include the steps of: acquiring real-time video data from a video collection device; detecting an object from the acquired video data; analyzing the behavioral pattern of the detected object to identify abnormal behavior, and generating a security event when the identified abnormal behavior satisfies at least one of a predetermined condition for suspected theft or a condition for non-payment within a time period; and storing object information related to the abnormal behavior in a blockchain.

[0009] In one embodiment, the step of detecting an object from the acquired image data may include the step of detecting an object corresponding to any one of a person, hand, product, checkout counter, or door in the real-time image data by utilizing a deep learning-based object detection model.

[0010] In one embodiment, the step of identifying abnormal behavior by analyzing the behavioral pattern of the detected objects includes the step of identifying abnormal behavior by inputting location and interaction information according to the time order of the detected objects into a time series analysis model, and the time series analysis model may include at least one of LSTM (Long Short-Term Memory) or TCN (Temporal Convolutional Network).

[0011] In one embodiment, the step of storing object information related to the abnormal behavior in a blockchain may include: a step of converting face feature or body feature information included in the object information into a high-dimensional feature vector through a deep learning-based embedding model; and a step of storing a hash value generated by applying a predefined hash function to the converted feature vector and a timestamp included in the object information together in the blockchain.

[0012] In one embodiment, the step of identifying abnormal behavior by analyzing the behavioral pattern of the detected object, and generating a security event when the identified abnormal behavior satisfies at least one of a predetermined condition of suspected theft or a condition of non-payment within a time period, may include: a step of obtaining information on the time of occurrence of the behavior and location information of the object identified as abnormal behavior; a step of obtaining transaction information for the corresponding time period from a payment system of a store in consideration of the information on the time of occurrence of the behavior; a step of determining whether a product presumed to have been acquired by the object has been successfully paid for by utilizing the information on the time of occurrence of the behavior and location information of the object and the obtained transaction information; and a step of restricting the generation of the security event when the successful payment of the product is confirmed.

[0013] The above-described means for solving the problem are merely exemplary and should not be interpreted as intended to limit the invention. In addition to the exemplary embodiments described above, additional embodiments may exist in the drawings and the detailed description of the invention. Effects of the invention

[0014] According to the means for solving the problem of the present invention described above, by extracting object behavior information from real-time video data and classifying abnormal behavior through a time-series-based artificial intelligence model, theft can be detected in real time, thereby enabling preemptive crime response.

[0015] Furthermore, by hashing facial or motion feature data included in abnormal behavior information and storing it on a blockchain, the present invention can generate tamper-proof theft records while securely protecting personal information, thereby enabling a region-based joint response to repetitive crimes through distributed blacklist sharing among stores and a function for re-detecting the same person.

[0016] However, the effects obtainable from the present invention are not limited to those described above, and other effects may exist. Brief explanation of the drawing

[0017] FIG. 1 is a block diagram of a computing device for providing a blockchain-based abnormal behavior analysis and information management method according to one embodiment of the present invention. FIG. 2 is a schematic flowchart of a blockchain-based abnormal behavior analysis and information management method according to one embodiment of the present invention. FIG. 3 is a diagram illustrating the operation of acquiring real-time video data from a video collection device installed in a self calculator according to an embodiment of the present invention and analyzing abnormal behavior. FIG. 4 is a diagram illustrating the operation of detecting an object from acquired image data and analyzing abnormal behavior according to an embodiment of the present invention. FIG. 5 is a diagram schematically illustrating the operation flow of a computing device when abnormal behavior is detected in a store according to one embodiment of the present invention. FIG. 6 is a diagram schematically illustrating the operation flow of a computing device when a past abnormal behavioral person re-enters a store according to an embodiment of the present invention. FIG. 7 is a diagram illustrating the configuration and utilization operation of a crime information blockchain according to an embodiment of the present invention. Specific details for implementing the invention

[0018] Embodiments of the present invention are described below with reference to the attached drawings so that those skilled in the art can easily implement the invention. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly explain the present invention in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the specification are denoted by similar reference numerals.

[0019] Throughout the specification of the present invention, when a part is described as being "connected" to another part, this includes not only cases where they are "directly connected," but also cases where they are "electrically connected" or "indirectly connected" with other elements interposed between them.

[0020] Throughout the specification of the present invention, when a member is described as being located "on," "on the upper," "on the top," "under," "on the lower," or "on the bottom" of another member, this includes not only cases where the member is in contact with the other member, but also cases where another member exists between the two members.

[0021] Throughout the specification of the present invention, when a part is described as "comprising" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0022] Throughout the specification of the present invention, some of the operations or functions described as being performed by a terminal, device, or device may instead be performed by a server connected to said terminal, device, or device. Likewise, some of the operations or functions described as being performed by a server may also be performed by a terminal, device, or device connected to said server.

[0023] Throughout the specification of the present invention, the term "at least one" may be defined as a term including both singular and plural forms, and it will be obvious that even if the term "at least one" is not present, each component may exist in a singular or plural form and may mean singular or plural. Furthermore, whether each component is provided in a singular or plural form may be changed according to the embodiment.

[0024] FIG. 1 is a schematic block diagram of a computing device for analyzing artificial intelligence-based abnormal behavior according to one embodiment of the present invention.

[0025] The method for analyzing artificial intelligence-based abnormal behavior described below can be performed by a computing device (100). For example, the computing device (100) may include a processor (110), a memory unit (120), and a network unit (130). However, the configuration of the computing device (100) is not limited thereto.

[0026] According to one embodiment of the present invention, the processor (110) may include a processor for data analysis and deep learning, such as a central processing unit (CPU) of a computing device, a general purpose graphics processing unit (GPGPU), or a tensor processing unit (TPU). The processor (110) may process a plurality of data stored in a memory unit (120) or a plurality of data obtained from a network unit (130).

[0027] According to one embodiment of the present invention, the memory unit (120) may include at least one storage medium among a magnetic medium such as a hard disk, a floppy disk and a magnetic tape, an optical recording medium such as a CD-ROM and a DVD, a magneto-optical medium such as a floptical disk, and a ROM, a RAM, and a flash memory. However, the memory unit (120) is not limited thereto.

[0028] According to one embodiment of the present invention, the network unit (130) may include a 3GPP (3rd Generation Partnership Project) network, an LTE (Long Term Evolution) network, a WIMAX (World Interoperability for Microwave Access) network, the Internet, a LAN (Local Area Network), a Wireless LAN (Wireless Local Area Network), a WAN (Wide Area Network), a PAN (Personal Area Network), a Bluetooth network, a NFC (Near Field Communication) network, a satellite broadcasting network, an analog broadcasting network, a DMB (Digital Multimedia Broadcasting) network, etc., but is not limited thereto and may include various wired / wireless communication networks.

[0029] According to one embodiment of the present invention, a computing device (100) can be connected to a user terminal via a network.

[0030] The computing device (100) may provide a web page, an app page, a program, or an application to a user terminal possessed by a user utilizing the computing device (100), thereby enabling the user to use the services provided by the computing device (100). For convenience of explanation, the program, application, service, and platform provided by the computing device (100) will be referred to as the present program, the present app, the present service, and the present platform, respectively.

[0031] The computing device (100) may be a device that can be implemented, for example, in the form of a program or application (app) installed on a user terminal. Alternatively, the method provided through the computing device (100) may be implemented in the form of a program or application and provided to the user through the user terminal. However, it is not limited thereto, and as another example, the computing device (100) may be provided in the form of a server capable of transmitting and receiving data with the user terminal. The computing device (100) provided in the form of a server may control the operation (for example, screen display) of the user terminal connected to the program or app provided by the computing device (100).

[0032] A user terminal may refer to a terminal possessed by a user who uses a computing device (100). The user may be referred to as a customer, a user, etc., and may be anyone regardless of gender or age. The user may use the service with or without installing the program or app by using the user terminal, and may also use the service by accessing the program or app to sign up or without signing up.

[0033] User terminals may include, but are not limited to, all types of wired and wireless communication devices such as, for example, PCS (Personal Communication System), GSM (Global System for Mobile communication), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (WCode Division Multiple Access), Wibro (Wireless Broadband Internet) terminals, smartphones, smartpads, tablet PCs, laptops, wearable devices, desktop PCs, etc.

[0034] The computing device (100) can transmit and receive data by being linked with the user terminal and the network unit (130).

[0035] The network section (130) may include, for example, a 3GPP (3rd Generation Partnership Project) network, an LTE (Long Term Evolution) network, a WIMAX (World Interoperability for Microwave Access) network, the Internet, a LAN (Local Area Network), a Wireless LAN (Wireless Local Area Network), a WAN (Wide Area Network), a PAN (Personal Area Network), a Bluetooth network, a NFC (Near Field Communication) network, a satellite broadcasting network, an analog broadcasting network, a DMB (Digital Multimedia Broadcasting) network, but is not limited thereto and may include various wired / wireless communication networks.

[0036] The present invention can identify abnormal behavior by acquiring real-time video from a video collection device to detect objects and analyze behavioral patterns. Furthermore, the present invention can determine abnormal behavior satisfying predefined conditions, such as theft or fraud, in real time by inputting the time, location, and interaction information of the detected objects into a time-series analysis model. Additionally, the present invention can verify payment status by analyzing transaction data from store payment systems, such as POS systems and self-checkout counters, and comparing the time and location information of an object's behavior occurrence with payment records. Moreover, the present invention can prevent tampering by embedding and hashing facial and physical feature information of objects identified as exhibiting abnormal behavior and securely storing it on a blockchain. Furthermore, the present invention can utilize the stored blockchain data to securely share abnormal behavior history information among multiple stores and identify repeat offenders. Additionally, the present invention can generate and transmit various forms of security events, such as administrator terminals, warning sounds, and notifications from external security agencies, according to risk levels. Moreover, the present invention can perform real-time analysis and long-term data storage in parallel by linking an edge device with a cloud server.

[0037] FIG. 2 is a schematic flowchart of a blockchain-based abnormal behavior analysis and information management method according to one embodiment of the present invention.

[0038] Referring to FIG. 2 according to one embodiment of the present disclosure, a blockchain-based abnormal behavior analysis and information management method may include the steps of: acquiring real-time video data from a video collection device (S10); detecting an object from the acquired video data (S20); identifying abnormal behavior by analyzing the behavioral pattern of the detected object, and generating a security event when the identified abnormal behavior satisfies at least one of a predetermined condition of suspected theft or a condition of non-payment within a time period (S30); and storing object information related to the abnormal behavior in a blockchain (S40).

[0039] According to one embodiment of the present disclosure, a computing device (100) can acquire real-time video data from a video acquisition device (S10). For example, the “video acquisition device” refers to a shooting device installed to acquire video of a surveillance target area in real time, and may include a fixed CCTV, a rotating PTZ (Pan-Tilt-Zoom) camera, a network camera, an ultra-wide-angle lens camera, an infrared (IR) camera, or a combination thereof. Additionally, the video acquisition device may be configured by combining a single type or multiple types of cameras depending on the store environment, installation location, and lighting conditions, and may include a variable focus lens, automatic zoom control, and day / night switching function as needed.

[0040] For example, “real-time video data” refers to continuous video frame data transmitted when the delay time between the time of shooting and the time of analysis is below a predefined standard. At this time, the computing device (100) can optimize video processing efficiency by dynamically adjusting the frame rate, resolution, and region of interest (ROI) according to network conditions, congestion, changes in illumination, etc., and can also perform real-time masking or blurring on unnecessary areas (e.g., keypad, area where personal information may be exposed).

[0041] For example, the video acquisition device may be connected to a computing device (100) via a network. Additionally, video data acquired from the video acquisition device may be transmitted to an object detection module and used to recognize objects such as people, hands, products, and checkout counters. In this process, the video acquisition device (camera) may switch not only to a fixed full-screen shooting mode but also to an event-based PTZ control mode, so that when the movement of a specific object is detected, the corresponding area may be automatically zoomed in and captured. Furthermore, the video timestamp and shooting location information may be combined with a behavior pattern analysis module and a transaction data linkage module to provide basis information for comparing and analyzing the time of a specific object's action with the time of payment within the store.

[0042] For example, in an unmanned convenience store environment, a 4K resolution network CCTV installed on the ceiling films the entire store at a speed of 30 frames per second, and the filmed video is streamed to an object detection program installed on a local edge server. The real-time video data includes the store entrance, product display shelves, and self-checkout areas, and the customer's movement path and behavior can be recorded throughout the entire area. For example, the computing device (100) can utilize a video processing algorithm in crowded situations or times of large changes in illumination to perform preprocessing such as noise removal, contrast enhancement, and color correction, and maintain the accuracy of object detection and behavior analysis.

[0043] Meanwhile, the computing device (100) can reliably secure high-quality real-time video required for analysis from the video collection device, thereby improving the accuracy of object detection and behavior analysis. In addition, since timestamps and location information are transmitted together, linked analysis with transaction data becomes possible, which can increase the reliability of verifying payment status and identifying suspected theft. Furthermore, by supporting various types of cameras, a flexible system configuration tailored to the store environment or security level is possible. Additionally, the connection between the camera and the computing device (100) can be established via a local network, a dedicated network, or a 5G-based wireless network, and in the event of a network failure, the edge device can prevent video loss by temporarily storing the video in a local buffer and then performing delayed transmission upon recovery.

[0044] According to one embodiment of the present disclosure, a computing device (100) can detect an object from acquired image data (S20). According to one embodiment, the computing device (100) can detect an object corresponding to any one of a person, hand, product, checkout counter, or door in the real-time image data by utilizing a deep learning-based object detection model. For example, a “deep learning-based object detection model” refers to an artificial intelligence model that simultaneously calculates the presence and location information of a specific object from an input image, and may include a Convolutional Neural Network (CNN), a Vision Transformer, or a hybrid structure thereof. For instance, the deep learning-based object detection model may be in a form that has been pre-trained and then fine-tuned to suit a specific store environment, and multi-scale feature extraction and data augmentation techniques may be applied to respond to changes in the appearance, size, color, and pose of the object.

[0045] For example, the objects to be detected include at least one of a person, a hand, a product, a checkout counter, or a door. Object detection refers to the process of identifying specific objects (e.g., people, hands, products, checkout counters, doors, etc.) present within an image frame and calculating location information of said objects. Location information can generally be expressed in the form of a bounding box, polygon mask, or keypoint coordinates in a 2D coordinate system, and may be converted into 3D coordinates by combining depth sensor or stereo camera data if necessary. For instance, object detection can be performed using CNN-based detectors such as YOLO (You Only Look Once), Faster R-CNN, or EfficientDet, or through Transformer-based vision models.

[0046] For example, the location, size, and class information of each object calculated in the object detection step can be transmitted to step S30 and used to analyze movement paths and interactions in chronological order. Additionally, among the detected objects, objects corresponding to products can be linked with transaction data and become subjects for payment verification. Furthermore, the object identification results can also be utilized to generate metadata to be recorded in blockchain storage. Meanwhile, object detection can be combined with a tracking module between time series frames as well as single-frame analysis, thereby reliably securing the continuous movement trajectory of the same object.

[0047] For example, the computing device (100) can receive real-time video frames using a YOLOv8-based object detection model and display objects classified into classes such as 'person', 'product', and 'checkout counter' as bounding boxes. At the moment a specific customer picks up a product, the bounding box coordinates and class of the product are recorded, and the relationship with the customer's hand position can be stored together as analysis data. Through this, the computing device (100) can determine the action sequence of 'picking up product - moving to checkout counter - payment' in a subsequent step. Additionally, the computing device (100) can maintain detection stability even in situations of occlusion between objects by applying a Vision Transformer-based attention mechanism even in scenes where multiple objects are mixed.

[0048] Meanwhile, the computing device (100) can provide base data for behavior analysis by detecting objects from acquired video data. In particular, the same algorithm can be applied to various shooting devices, such as PTZ cameras and drone cameras, as well as fixed CCTVs, so there are few restrictions on the installation environment. In addition, the computing device (100) can maintain high detection accuracy even when customer congestion or obstruction of view occurs by utilizing a high-precision deep learning model. Furthermore, the computing device (100) can maintain real-time performance by combining a lightweight object detection model (Tiny model) with an edge computing device, even in low-resolution environments or environments with limited network bandwidth.

[0049] According to one embodiment, the computing device (100) may apply a Vision Transformer-based attention mechanism to reduce detection errors caused by occlusion between objects in a crowded scene where multiple objects exist simultaneously.

[0050] According to another embodiment of the present disclosure, a computing device (100) can divide an input image frame into patches of a certain size, convert each patch into a high-dimensional embedding vector, and then perform a Multi-Head Self-Attention operation to learn long-term and short-term dependencies between patches. At this time, the attention weights are learned to focus not only on the area around the bounding box of an object but also on a part of the patch of the occluded object, so that the object can be accurately reconstructed even in a partially occluded state.

[0051] For example, if a customer passes in front of another customer near a store entrance and is partially obscured, the Vision Transformer's attention mechanism can recognize the obscured object as the same object by combining the appearance features of that customer (clothing color, body proportions, etc.) acquired in the previous frame with the features of the unobscured area in the current frame.

[0052] In addition, during this process, the computing device (100) can combine a CNN-based local feature extraction module and a Transformer-based global context analysis module in a hybrid form to simultaneously reflect the relationship between detailed texture information and long-range patches. With this configuration, the computing device (100) can maintain the stability of object detection and the continuity of tracking even when partial occlusion or overlapping occurs, even in crowded scenes where multiple customers, products, and checkout counters are included in the same image. For example, even if the occlusion period lasts for a certain period of time or longer, the computing device (100) can improve the accuracy of object ID re-identification (Re-ID) by utilizing context information from the preceding and succeeding frames by applying the Temporal Transformer, a time-series extension structure of the Transformer.

[0053] Referring to FIG. 4 for an example, a computing device (100) can detect areas corresponding to the face (1), hand (2), and product (3) of an object in the acquired real-time image data. For example, at the moment a first customer picks up a product (3) with their hand (2) near a self-checkout counter in a store, another second customer may move across the front of the first customer, and part of the first customer's hand (2) may be obscured by the second customer's body. In such a case, a traditional CNN-based object detection model may fail to recognize part or all of the hand (2) or classify it as an incorrect object.

[0054] In one embodiment, the computing device (100) can reliably recognize the presence of the hand (2) by applying a Vision Transformer-based attention mechanism and combining the location of the hidden hand (2) with features extracted from the previous frame (skin color, hand shape, relative positional relationship with surrounding products (3), etc.) and some patch features of the hand (2) exposed in the current frame. In particular, the computing device (100) can analyze the spatial correlation between the hand (2) and the product (3) through Multi-Head Self-Attention and reconstruct the original behavior pattern by interpolating the patches of the hidden area. For example, in a scenario where a first customer picks up a product (3) with their hand (2) and moves toward the checkout counter, the system can simultaneously track the trajectory of the hand (2) and the movement trajectory of the product (3), and even if the hand (2) is partially hidden along the way, maintain it as the same object to completely record the behavior sequence of 'picking up product - moving - payment'. In this way, according to the present embodiment, detection stability and tracking continuity can be maintained by utilizing the correlation between the face (1), hand (2), and product (3) even in crowded scenes or occlusion situations where the movement paths of multiple customers overlap. Therefore, errors in abnormal behavior analysis caused by partial occlusion can be reduced, and the reliability of determining suspected theft conditions can be improved.

[0055] According to one embodiment of the present disclosure, a computing device (100) can identify abnormal behavior by analyzing the behavioral pattern of a detected object, and if the identified abnormal behavior satisfies at least one of a predetermined condition of suspected theft or a condition of non-payment within a time period, it can generate a security event (S30).

[0056] For example, a computing device (100) can identify abnormal behavior by analyzing the time-series coordinates, movement speed, movement direction, whether it is stopped, and changes in distance between objects of an object detected in step S20. At this time, the abnormal behavior may include patterns involving theft, fraudulent payment, insider theft, or collaborative crime compared to normal shopping or work behavior. For example, it may include 'moving via an abnormal path after acquiring a product,' 'putting a product inside a hidden object,' 'bypassing the checkout counter,' 'delivering goods,' etc. The above analysis may be performed through a time-series analysis model having a structure of LSTM (Long Short-Term Memory), TCN (Temporal Convolutional Network), or a hybrid thereof. LSTM learns behavioral sequences by considering long-term dependency, and TCN can provide high precision compared to single-frame-based detection by simultaneously analyzing changes in short intervals and long-term context.

[0057] For example, the computing device (100) can determine whether there is abnormal behavior by comparing it with predefined conditions for suspected theft or conditions for non-payment within a certain time based on identified behavioral patterns. For example, conditions for suspected theft may include concealment in a bag or clothing, bypassing the checkout counter, concealment of money by an internal employee, collaboration of two or more people (covering, transferring, etc.). For example, if the identified abnormal behavior satisfies the conditions for suspected theft, the computing device (100) can generate a security event and transmit it to an administrator terminal, a security server, or a cloud notification system. As another example, the computing device (100) can generate a security event even if the identified abnormal behavior satisfies the conditions for non-payment within a certain time, and if it satisfies both conditions for suspected theft and non-payment simultaneously, it can classify it as a priority warning target by increasing the risk weight. For example, conditions for non-payment within a certain time may include failure to check POS or self-checkout payment logs within a certain time (e.g., 3 minutes) after acquiring the product. For example, if a ‘hand’ object picks up a ‘product’ object and moves into a ‘bag’ object, and subsequently the product object is no longer detected and there is no payment log in that time period, the computing device (100) can determine this as a bag concealment pattern and generate a security event. Additionally, if the conditions of suspected theft and non-payment within a time period are satisfied simultaneously, the system can increase the risk weight of the event to raise the priority of the administrator notification.

[0058] According to one embodiment of the present disclosure, a computing device (100) can obtain behavior occurrence time information and location information of an object identified as abnormal behavior. Here, the behavior occurrence time information may be calculated based on the frame timestamp of real-time video data obtained from a video collection device. Additionally, the location information may include 2D coordinates or 3D spatial coordinates extracted from the object detection result at the corresponding time. If there is only one object, the computing device (100) can directly extract time and location information corresponding to a single tracking ID of the object, and if there are multiple objects, the computing device can process multiple behavior events that occurred at the same time in parallel by individually managing time-series location data in conjunction with the tracking ID assigned to each object.

[0059] In addition, since blind spots may exist depending on the store structure or shooting angle, the computing device (100) may apply a multi-camera-based coordinate calculation technique using multiple image acquisition devices. Specifically, the computing device (100) stores the camera intrinsic parameters and camera extrinsic parameters of each image acquisition device as pre-calibration data, and can precisely calculate 3D spatial coordinates by matching 2D coordinates detected from multiple camera views for the same object and performing a triangulation or stereo vision algorithm. The computing device (100) can utilize the calculated 3D location information to map to the zone coordinate system inside the store, thereby accurately determining where the object is located, such as on a shelf, checkout counter, or exit gate. For example, if the exit camera and the display case camera photograph the same object, the computing device (100) can triangulate the two-dimensional coordinates obtained from the two cameras to verify with high precision that the object is actually located in the exit gate zone.

[0060] For example, the computing device (100) can obtain transaction information for the corresponding time period by linking in real-time with the store's payment system (POS, self-checkout counter, etc.) based on the behavior occurrence time information and location information obtained in this way. The transaction information may include the payment time, product code, payment amount, and payment terminal location, and the linking method may include one or more of a direct connection to a local network, a cloud API-based linkage (REST API, WebSocket, gRPC), or a message broker-based real-time event reception (Kafka, MQTT, RabbitMQ). Subsequently, the computing device (100) can determine whether the product presumed to have been acquired by the object was successfully paid for by mapping the location information where the payment terminal is installed with the installation location information of the video collection device and comparing the zone where the object is located with the zone where the payment act took place. For example, if a specific object exhibits abnormal behavior near the exit gate, but the corresponding product code is recorded in the payment log at the payment counter adjacent to the exit at the same time, it can be determined as a normal payment.

[0061] Additionally, the computing device (100) can minimize unnecessary alerts by restricting or suppressing the generation of security events regarding abnormal behavior when normal payment of a product is confirmed. On the other hand, if no payment log exists as a result of comparison with transaction information, the computing device (100) can immediately generate a security event and, if necessary, transmit the movement path, identification image, and acquired product information of the object to the administrator terminal. Through this configuration, the present invention goes beyond simple behavior detection and combines multi-camera-based precise location calculation and payment verification to reduce false positive rates and significantly improve the accuracy of response to actual theft situations.

[0062] According to one embodiment of the present disclosure, a computing device (100) can identify abnormal behavior in the form of collaboration by analyzing the interaction patterns between the objects in a time-series manner based on the time information and location information of the occurrence of behavior of a plurality of objects.

[0063] For example, the computing device (100) can detect collaborative crime patterns that are difficult to identify as the sole abnormal behavior of individual objects, such as when a first object attempts to acquire a product from a specific product display stand in a store, and a second object moves its body in front of it to block the CCTV view, or when the first object secretly transfers the acquired product into a third object's shopping cart or bag. To this end, the computing device (100) can maintain tracking IDs for each object and classify cases where two or more objects exist in close proximity at the same time and one of the objects performs a predefined interaction pattern, such as “blocking the view” or “transferring a product,” as abnormal behavior candidates. Whether such interaction occurs can be determined by combining the change in the object's position coordinates, the intersection over union (IoU) ratio of the bounding box, and the motion sequence pattern of the pose estimation result.

[0064] According to another embodiment of the present disclosure, a computing device (100) can detect the possibility that an object performing abnormal behavior is an internal employee. To this end, the computing device (100) stores identification information of an internal employee registered in advance (e.g., employee ID, employee number, face feature vector, uniform or badge recognition result) in a database and performs matching with the corresponding features in a real-time object detection and identification step. For example, the computing device (100) can recognize a uniform or name tag worn by an object using a face recognition model (FaceNet, ArcFace, etc.) or a clothing pattern recognition model (CNN-based clothing classifier), and if the matching score is greater than or equal to a threshold value, the object can be identified as an internal employee. Additionally, if a pattern is detected in which an object identified as an internal employee fails to scan products during the payment step or hides cash after opening the cash drawer of a POS terminal, the computing device (100) can classify this as a suspected internal theft event. At this time, the internal employee ID and the corresponding abnormal behavior event can be combined and stored in a separate security log, and can be transmitted to the management server in real time if necessary.

[0065] According to one embodiment, a computing device (100) can calculate the risk level of an abnormal behavior event and increase the risk level weight when certain conditions are met. To calculate the risk level, the computing device (100) may assign a basic risk score by synthesizing the type of behavior (e.g., acquiring goods, unauthorized passage through a checkout counter, blocking of view, etc.), the number of related objects (single or two or more collaborating), the location where the event occurred (exit, checkout counter, blind spot, etc.), and the history of abnormal behavior of the same object in the past. Additionally, the computing device (100) can increase the risk level weight if one or more of the following conditions are met: simultaneously satisfying the conditions of suspected theft and non-payment within a time limit; identifying the object performing the abnormal behavior as an internal employee; or simultaneously performing abnormal behavior by multiple objects of the same group. Additionally, the computing device (100) can utilize the increased risk score to reflect it in the priority field of a security event queue and place it at the top of the administrator notification list, thereby automatically increasing the frequency and intensity of notifications through multiple channels, such as alarm sounds, push notifications, and SMS transmissions, as the risk level increases.

[0066] According to one embodiment of the present disclosure, a computing device (100) may store object information related to abnormal behavior in a blockchain (S40). For example, the object information may include the time at which the abnormal behavior occurred, unique identification information of the object (e.g., facial features, body features, clothing color, gait pattern, etc.), an identifier of a product that the object interacted with, the type of abnormal behavior, location information of occurrence, and an identification hash value of a related image. For instance, to protect personal information, the computing device (100) may convert facial features or body features among the object information into high-dimensional feature vectors (e.g., 128 dimensions, 512 dimensions) using a deep learning-based embedding model (FaceNet, ArcFace, ReID, etc.), and then apply a predefined hash function (SHA-256, SHA3, BLAKE2, etc.) to generate a hash value that cannot be restored from the original. The generated hash value may be combined with behavior occurrence time information (timestamp), abnormal behavior type code, and store code to form a single transaction and may be stored in a blockchain.

[0067] For example, the blockchain structure may include at least one of a public blockchain (Ethereum, Hyperledger Besu, etc.), a permissioned blockchain (Hyperledger Fabric, Quorum, etc.), or a hybrid blockchain structure. All data recorded on the blockchain is digitally signed to prevent tampering, and its reliability is ensured through a multi-node verification process. Blockchain nodes can be distributed across store servers, headquarters servers, regional security integrated control servers, etc., thereby maintaining integrity even in the face of single-server hacking or attempts to tamper with data.

[0068] For example, the information stored on the blockchain can include not only simple event data but also behavioral context information before and after abnormal behavior. For instance, by sequentially recording sequence data such as T1: coordinates and timestamp at the time of product acquisition, T2: access to the checkout counter and route information, T3: time of access to the store entrance, and T4: payment log status, it is possible to reconstruct events based on the flow of time rather than static detection results from a single frame.

[0069] In addition, the blockchain network provides a blacklist sharing function between stores. To prevent the exposure of personal information, each store stores only the hash values ​​of embedding vectors for facial and physical features, and when querying the blacklist, it determines whether the individuals are the same by comparing whether the hash values ​​match. To achieve this, Zero-Knowledge Proof (ZKP) technology is applied to prove that the hash value was derived from a specific object without disclosing the original data, and homomorphic encryption is utilized to enable comparison operations of encrypted data on the blockchain without decryption.

[0070] For example, if a suspected theft event occurs at a specific store and the theft is confirmed because there is no POS payment log, the computing device (100) generates the embedding hash of the object, the time of occurrence, the store code, and the type of abnormal behavior as a transaction and records it on the blockchain, and simultaneously shares it in real time with other stores on the same network, thereby preventing the same person from attempting abnormal behavior again at another store.

[0071] Accordingly, according to the present embodiment, the computing device (100) provides a secure and privacy-protected structure that goes beyond detecting abnormal behavior within a single store and can safely share and utilize the history of incidents and behavioral patterns between multiple stores, thereby expanding the scope of crime prevention and ensuring data reliability that can be used as legal evidence while complying with global personal information protection regulations.

[0072] FIG. 3 is a diagram illustrating the operation of acquiring real-time video data from a video collection device installed in a self calculator according to an embodiment of the present invention and analyzing abnormal behavior.

[0073] According to one embodiment of the present disclosure, a computing device (100) can acquire image data in real time from an image acquisition device included in a self-checkout counter and monitor and analyze the behavior of an object occurring around the counter in real time. Referring to FIG. 3, the self-checkout counter may include a first area (10), a second area (20), and a third area (30).

[0074] For example, the first area (10) is an area where the user temporarily places items before payment, and the image collection device can record the condition of the items before purchase by photographing the quantity, size, shape, barcode status, etc. of the items located in the area. The second area (20) includes a display device and an image collection device that interact with the user, and the image collection device can photograph the user's upper body, hand position, movement of the items, etc. The third area (30) is an area where items for which barcode scanning has been completed are placed, and the image collection device can be used to determine whether the payment process is normal by tracking the process of moving the items to the area.

[0075] For example, the computing device (100) can determine in real time whether the item has successfully undergone barcode recognition and payment steps by analyzing the item movement path and time information between the first area (10) and the third area (30), and by combining and analyzing the user's hand movement and item manipulation motion patterns captured in the second area (20). Through this, if the item moves from the first area to the third area or toward the entrance without payment, it can be identified as an abnormal behavior of non-payment and a security event can be generated.

[0076] For example, a video collection device may be installed at the top, side, or external point of a self-checkout machine to secure a forward view that includes the areas for inserting items, payment, and packaging. A computing device (100) may acquire real-time video data from the video collection device and then apply an object detection and tracking algorithm to identify objects such as customers, internal staff, and moving goods. Additionally, regarding the identified objects, the computing device (100) may determine whether there is abnormal behavior through motion sequence analysis and time-series coordinate data processing. For example, this applies to cases where a customer puts an item into a bag without going through the payment process after acquiring it, or where an internal staff member hides money or goods. Subsequently, the identified abnormal behavior may be stored in a blockchain network along with the identification information of the object (e.g., facial features, body features, clothing color, gait pattern), the type of abnormal behavior, the location and time of occurrence, and the hash value of the related video. If necessary, the information may be shared in real-time with other stores participating in the same blockchain network to prevent the same person from attempting to repeat the behavior in advance. Additionally, the computing device (100) may identify collaboration patterns between multiple objects (e.g., blocking of view + acquisition of goods, delivery of goods, etc.) or specific patterns of internal employees (e.g., skipping of payment steps, concealment of money, etc.), calculate the risk level of the event, and adjust the priority of the manager notification by increasing the risk level weight when the conditions of suspected theft and non-payment are simultaneously met.

[0077] FIG. 4 is a diagram illustrating the operation of detecting an object from acquired image data and analyzing abnormal behavior according to an embodiment of the present invention. Referring to FIG. 4, the computing device (100) can perform detailed feature recognition of an object and interaction analysis to analyze the customer's behavior more finely.

[0078] For example, referring to FIG. 4, the self calculator may include a first area (10), a second area (20), and a third area (30). The computing device (100) may acquire real-time image data in frames from a plurality of image acquisition devices corresponding to each area. Subsequently, the computing device (100) may apply an object detection algorithm to each acquired image frame to identify objects of interest such as a user's face, upper body, both hands, items being carried, or a packaging bag, and may extract the bounding box coordinates of the objects on a frame-by-frame basis.

[0079] The computing device (100) can sort detected object information into a time series and generate a continuous trajectory of the same object using an object tracking algorithm. Additionally, the computing device (100) can analyze behavioral characteristics such as position change, movement speed, motion trajectory, interaction targets, and dwell time of each object. For example, in the first area (10), the quantity, size, shape, and whether a barcode is attached to items temporarily placed before payment can be recorded, and patterns of the items staying for a long time or suddenly disappearing can be tracked. Additionally, in the second area (20), the computing device (100) can analyze the user's hand trajectory and movements around the payment interface to verify whether a barcode scan motion has been performed, whether a payment button has been touched, and whether a payment method has been inserted. Additionally, in the third area (30), the computing device (100) can determine whether items are moved and loaded normally onto a packaging rack after barcode scanning is completed, whether unscanned items are moved directly to a packaging bag, or whether only some items are paid for.

[0080] Based on the analysis of behavioral patterns, if an item moves directly to a third area without a barcode scanning operation, if a line of sight blocking operation and an item concealment operation occur consecutively, or if payment is not completed within a preset time, the computing device (100) can identify such behavior as abnormal behavior. If the abnormal behavior satisfies one or more of the conditions for suspected theft or non-payment within the time, the computing device (100) can generate a security event including a video clip, object ID, behavioral characteristics, location of occurrence, and time information at that time. Additionally, the computing device (100) can utilize this for post-audit and evidence acquisition by recording it on a blockchain network to prevent forgery and tampering.

[0081] According to another embodiment of the present disclosure, a computing device (100) can track a movement path by extracting only feature information of the foot area, rather than the entire body information of an object in which abnormal behavior has been identified. To this end, only the lower 20-30% area of ​​the bounding box obtained in the object detection step is set as the Region of Interest (ROI), and feature vectors such as shoe shape, color, stride length, and walking pattern can be calculated in that area.

[0082] The above foot feature vector is converted into a high-dimensional vector through a deep learning-based embedding model and then continuously tracked through a multi-frame feature matching algorithm. In this embodiment, compared to methods that track the entire body or face area, the amount of computation can be reduced by more than 50%, which enables real-time processing, particularly in Edge AI devices or low-spec GPU environments. Additionally, since foot tracking works effectively even when the upper body is obscured by a mask, hat, etc., the false positive rate caused by hidden objects can be reduced compared to face-based tracking. The tracked foot information includes data such as the object's movement trajectory coordinates, movement speed, and movement direction, and by combining this with CCTV array information inside the store, the direction and location in which the object left the store can be estimated.

[0083] Meanwhile, in this embodiment, the tracked foot feature information is also recorded on the blockchain only with hashed feature values, without storing the original images, thereby ensuring the integrity of the tracking results while complying with personal information protection regulations. Through this, store security personnel can quickly identify the escape route immediately after an incident occurs and enable coordinated response with nearby stores or security personnel.

[0084] FIG. 5 is a diagram schematically illustrating the operation flow of a computing device when abnormal behavior is detected in a store according to one embodiment of the present invention.

[0085] Referring to FIG. 5 according to one embodiment of the present disclosure, a computing device (100) can receive video from a CCTV / IP camera and perform frame extraction to process each frame into an image. The image can be standardized into a preprocessed image that has undergone preprocessing, and the computing device (100) can perform face detection using the preprocessed image. The detected face (or full body of a person) can be input into an embedding model and converted into a fixed-length vector through embedding, wherein the vector can be serialized through normalization and quantization processes. Subsequently, the vector can be combined with an encryption key (e.g., device unique K_dev or session key K_sess) provided by a security module within the device to be prepared as protected data in the form of vector+encryption key, and after including a one-time random number (nonce), salt, and key ID in this protected data, at least one of HMAC-SHA-256, SHA-3, and BLAKE2 can be applied to convert it into an encryption hash. The result of the transformation may be a collision-resistant cryptographic hash, the original vector is immediately destroyed from memory (optional encrypted with AEAD and stored off-chain), and only the hash may be used for on-chain recording. The foregoing is merely one example and is not limited thereto.

[0086] For example, before the embedding vector is recorded on-chain, a nearest neighbor search based on Euclidean distance or cosine similarity may be performed on the historical vector index (off-chain feature vector repository) in the comparison analysis module, and if the calculated analysis value ≥ threshold (the threshold is pre-set by the operation policy and ROC curve), a same person tracking signal may be generated. If the same person is determined or re-identified, the corresponding matching result (distance value, top k candidates, confidence) may be combined into event metadata and passed to a subsequent stage.

[0087] Additionally, the computing device (100) can generate bounding boxes for each object by performing object detection, such as people and hands, in the frame image in parallel with the face / person embedding flow. Additionally, the computing device (100) can accumulate continuous bounding box data for each object connected in the order of frames as time-series data. The computing device (100) can perform time-series data construction with this data and can produce time-series features such as speed, acceleration, heading angle, dwell time, hand-product IoU change, scanner proximity / contact state, and transition between areas (1st area - 2nd area - 3rd area) through activity log extraction. The extracted features can be fed into an LSTM / TCN classifier and output as class probability values ​​(detection / prediction) for events such as theft, damage, or bypass payment. Model parameters can be periodically retrained through the mass collection (learning) of activity logs accumulated during operation, and if abnormal behavior is determined based on the analysis of the classifier's results, the event can be confirmed.

[0088] When abnormal behavior is confirmed, the computing device (100) can generate a transaction including an event ID, store code, camera ID, timestamp, location zone, class probability value, threshold, nearest neighbor analysis value, top candidate list, related product code, and the cryptographic hash (embedding hash and related video clip hash) and record it in the crime information blockchain database. The blockchain database may include at least one of permissioned, public, or hybrid types. For example, only hash / summary values ​​with minimized identifiability are stored on-chain, while the original video or vector is encrypted and stored in an off-chain storage, and the on-chain record can be used as an integrity anchor and reference key for off-chain resources. When the block is confirmed, a push alert notification may be sent to the security officer's terminal in the form of an APP message.

[0089] Additionally, a security officer or representative may send a request to view the video during an incident review. For example, the computing device (100) may allow the viewing of the video of the off-chain encrypted resource pointed to by the transaction after performing authorization based on an access policy recorded on the blockchain. The viewing log is then left on the chain as a summary hash to enable auditing, and the recurrence of the same person or similar pattern can be immediately verified by comparison with past chain data.

[0090] Referring to FIG. 5, the computing device (100) can organically perform the entire process of frame extraction → image preprocessing → embedding and vector protection (vector + cryptographic key, converted into cryptographic hash) → comparative analysis (analysis value ≥ threshold, Euclidean distance, cosine similarity) → tracking of identical person / determination of abnormal behavior (time series features, class probability value) → storage of crime information blockchain → push alert notification → video viewing request, authorization authentication, and video viewing in a manner that guarantees integrity and privacy.

[0091] FIG. 6 is a diagram schematically illustrating the operation flow of a computing device when a past abnormal behavioral person re-enters a store according to an embodiment of the present invention.

[0092] Referring to FIG. 6 according to one embodiment of the present disclosure, a computing device (100) can acquire video from a CCTV or IP camera. Additionally, the computing device (100) can extract frames from the video. The extracted frames can be converted into images, and face detection can be performed on the images. The detected face images can be converted into preprocessed images through a preprocessing process, and the preprocessed images can be converted into feature vectors through an embedding process.

[0093] The transformed feature vector can be combined with a pre-set cryptographic key to form vector and cryptographic key data, which can then be converted into a cryptographic hash. The generated cryptographic hash can be compared and analyzed with stored data on past abnormal behaviors. During this comparative analysis process, it can be determined whether the analysis value exceeds thresholds such as Euclidean distance or cosine similarity. If the value exceeds the threshold, the object can be presumed to be the same person, and the result can be stored in the form of a cryptographic hash on a crime information blockchain.

[0094] For example, the crime information blockchain may contain identification information regarding past abnormal behaviors (theft, vandalism, etc.) within the store, and if a new comparative analysis result determines that the abnormal behavior has re-entered, the information may be verified based on the blockchain and provided to the security officer. At this time, the computing device (100) may send a push alert notification to the representative / security officer via an APP message, and the security officer may check real-time video through a video viewing request after performing authorization based on the blockchain information.

[0095] In this way, the computing device (100) can improve the security of the store by automatically recognizing cases where a past perpetrator of abnormal behavior (theft, vandalism, etc.) re-enters the store, recording the relevant facts on the crime information blockchain, and simultaneously providing an immediate warning to the security officer.

[0096] FIG. 7 is a diagram illustrating the configuration and utilization operation of a crime information blockchain according to an embodiment of the present invention.

[0097] According to one embodiment of the present disclosure, a computing device (100) can detect abnormal behavior, such as theft or damage, by collecting video input from a CCTV or IP camera installed in a store. When such abnormal behavior is detected, the computing device (100) can automatically record the event in a crime information blockchain.

[0098] The computing device (100) can generate a new block whenever a new crime event is detected. The generated block can record crime information including an event ID, an encrypted hash value, a criminal code, and a crime type code. Additionally, the computing device (100) can store a store code, a manager ID, a timestamp, and an associated product code together to record management information. The block can be generated simultaneously at multiple stores, such as Store A through Store N, and the computing device (100) can register the block to a blockchain network if the store agrees to the management of crime information.

[0099] The computing device (100) can not only store crime information at the individual store level based on the generated blockchain, but also process it so that it is organically shared across the network. Therefore, the store security manager or representative can utilize the relevant statistical and analytical information provided by the computing device (100).

[0100] Specifically, the computing device (100) may perform analysis of abnormal behavior and crime by region based on data accumulated on the blockchain, risk warnings for regions and areas with a high number of recent abnormal behaviors, analysis of abnormal behavior by time of day, analysis by type of abnormal behavior (e.g., theft, internal employee crime, etc.), and analysis of related products (e.g., focusing on high-risk items such as cigarettes, liquor, and cash). In addition, the computing device (100) may provide analysis results in the form of urgent, weekly, and monthly reports to support managers in establishing store security strategies from a real-time and medium-to-long-term perspective.

[0101] In particular, the computing device (100) processes information recorded individually at the store level so that it is interconnected through a blockchain network, thereby enabling rapid warning and tracking when a person who exhibited abnormal behavior at a past store re-enters a new store. Accordingly, the computing device (100) can effectively contribute to preventing recidivism and strengthening store security.

[0102] In the description above, steps S10 to S40 may be further divided into additional steps or combined into fewer steps, depending on an embodiment of the present invention. Additionally, some steps may be omitted as necessary, and the order between steps may be changed.

[0103] A blockchain-based abnormal behavior analysis and information management method according to one embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either individually or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the present invention, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The above-described hardware device may be configured to operate as one or more software modules to perform the operation of the present invention, and vice versa.

[0104] In addition, the aforementioned blockchain-based abnormal behavior analysis and information management method can also be implemented in the form of a computer program or application executed by a computer stored on a recording medium.

[0105] The foregoing description of the present invention is for illustrative purposes only, and those skilled in the art will understand that other specific forms can be easily modified without altering the technical spirit or essential features of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single unit may be implemented in a distributed manner, and components described as distributed may likewise be implemented in a combined form.

[0106] The scope of the present invention is defined by the claims set forth below rather than by the detailed description above, and all modifications or variations derived from the meaning and scope of the claims and equivalent concepts thereof should be interpreted as being included within the scope of the present invention.

Claims

Claim 1 A blockchain-based abnormal behavior analysis and information management method performed by a computing device, comprising: a step of acquiring real-time video data from a video collection device; a step of detecting an object from the acquired real-time video data; a step of identifying abnormal behavior by analyzing the behavioral pattern of the detected object, and generating a security event if the identified abnormal behavior satisfies at least one of a predetermined condition for suspected theft and a condition for non-payment within a time period; The method includes the step of storing object information related to the above abnormal behavior in a blockchain, and the step of detecting objects from the acquired real-time video data includes the step of inputting the acquired real-time video data into a deep learning-based object detection model including a Convolutional Neural Network (CNN) model and a Vision Transformer model to detect multiple objects corresponding to people, hands, products, checkout counters, and doors, wherein, when the acquired real-time video data represents a crowded scene where the multiple objects exist simultaneously, the method further includes the step of applying an attention mechanism of the Vision Transformer model to reduce detection errors caused by occlusion between different objects, and the step of applying the attention mechanism of the Vision Transformer model includes the step of dividing the acquired real-time video data into patch units of a certain size for each video frame, converting each patch into a high-dimensional embedding vector, and then performing a Multi-Head Self-Attention operation on the converted high-dimensional embedding vector to learn long-term and short-term dependency relationships between each patch; and the Multi-Head Self-Attention operation so as to focus not only on the area around the object's bounding box but also on patches where the object is only partially exposed. The method comprises: a step of learning attention weights; and a step of reconstructing an object in a partially occluded state using the long-term and short-term dependencies between each of the learned patches and the learned attention weights, wherein the step of reconstructing an object in a partially occluded state comprises:In cases where the acquired real-time video data represents a crowded scene containing multiple people, multiple products, and multiple checkout counters within a single video frame, in order to maintain the stability of object detection and the continuity of tracking even if partial occlusion or overlapping occurs, the method includes a step of re-identifying an object whose occlusion persists for longer than a preset time by combining the output of the local feature extraction module of the Convolutional Neural Network (CNN) model and the output of the global context analysis module of the Vision Transformer model, and simultaneously utilizing texture information of patches included in the single video frame and distance relationships between patches; the step of identifying abnormal behavior by analyzing the behavioral patterns of the detected objects includes a step of identifying abnormal behavior by inputting location and interaction information according to the chronological order of the detected objects into a time-series analysis model, wherein the time-series analysis model includes a Long Short-Term Memory (LSTM) and a Temporal Convolutional Network (TCN); the location and interaction information according to the chronological order of the detected objects includes the time-series coordinates, movement speed, direction of movement, whether they are stopped, and changes in distance between objects; and the abnormal behavior is normal shopping behavior and work behavior The behavioral patterns include unanalyzed theft, fraudulent payment, insider theft, and collaborative crimes; the aforementioned predetermined suspected theft conditions include behavioral patterns of concealing goods in a bag, concealing goods in clothing, bypassing the checkout counter by grasping goods, concealing money by a checkout employee, and collaborative behavioral patterns in which two or more people conceal and pass goods; the aforementioned non-payment within a time condition includes failure to check POS and self-checkout payment logs within a certain time after grasping goods; and the step of generating the security event comprises the step of obtaining behavior occurrence time information and location information of the object identified as abnormal behavior; and, considering the behavior occurrence time information,A step of obtaining transaction information for the corresponding time period from the store's payment system; a step of determining whether a product identified as being held by the object has been successfully paid for by utilizing the object's behavior occurrence time information and location information and the obtained transaction information, while mapping the location information where the payment terminal is installed with the installation location information of the video collection device, and comparing the area where the object is located with the area where the payment act took place, in order to reduce false positive rates and improve the accuracy of response to actual theft situations by combining multi-camera-based precise location calculation and payment verification, going beyond simple behavior detection; and a step of restricting the generation of a warning sound when the successful payment of the product is confirmed. A blockchain-based abnormal behavior analysis and information management method characterized by comprising the step of generating a warning sound and transmitting the movement path of an object identified as holding the product, an image of the identified object, and information on the identified product to an administrator terminal when normal payment of the product is not confirmed, wherein the behavior occurrence time information is calculated based on the frame timestamp of real-time video data acquired from the video collection device, and the location information includes 2D coordinates or 3D spatial coordinates extracted from the object detection result at a point in time included in the behavior occurrence time information, wherein the behavior occurrence time information and the location information include time and location information corresponding to a single tracking ID of the object when there is only one object identified as holding the product, and when there are multiple objects identified as holding the product, time-series location data is managed individually in conjunction with the tracking ID assigned to each object so that multiple behavior patterns occurring at the same time are processed in parallel. Claim 2 delete Claim 3 delete Claim 4 A blockchain-based abnormal behavior analysis and information management method according to claim 1, wherein the step of storing object information related to the abnormal behavior in a blockchain comprises: a step of converting facial feature and body feature information included in the object information into a high-dimensional feature vector through a deep learning-based embedding model; and a step of storing a hash value generated by applying a predefined hash function to the converted feature vector and a timestamp included in the object information together in the blockchain. Claim 5 delete Claim 6 A blockchain-based abnormal behavior analysis and information management method, characterized in that, in claim 1, after the step of storing object information related to the abnormal behavior in a blockchain, it further includes the step of performing regional abnormal behavior and crime analysis, time-based abnormal behavior analysis, abnormal behavior type analysis, and related product analysis based on the data stored in the blockchain, and generating the results of the performed analyses into a report.

Images