Abnormal behavior handling method and device, equipment, storage medium and program product

Abstract

The invention provides an abnormal behavior handling method which can be applied to the technical field of big data and financial science and technology. The method comprises the steps that static attribute information of a target area and multi-modal data generated by the target area are obtained, the target area represents a physical area with service attributes, and the multi-modal data represents data generated by an entity object set associated with the service attributes in the target area; matching the static attribute information with the multi-modal data to generate state information of an entity object set in the target area; detecting the state information of the entity object set in the target area, and determining the confidence degree that the entity object set in the target area has an abnormal behavior; and processing the abnormal behavior based on a relationship between the confidence of the abnormal behavior in the entity object set and a preset confidence threshold. The invention further provides an abnormal behavior processing device and equipment, a storage medium and a program product.

Description

Technical Field

[0001] This application relates to the fields of big data and financial technology, and more specifically to a method, apparatus, device, storage medium, and program product for handling abnormal behavior. Background Technology

[0002] As financial institutions deepen their digital transformation and intelligent risk control requirements continue to rise, the need for monitoring and handling abnormal behavior in bank branch work areas is becoming increasingly complex and sophisticated.

[0003] Currently, monitoring of bank branch work areas faces the following problems: First, manual monitoring has limitations in efficiency, as monitoring personnel need to pay attention to multiple video feeds simultaneously, resulting in limited ability to identify abnormal behavior; second, the system response has timeliness issues, with a significant delay from the occurrence of an event to its resolution, making it difficult to meet the needs of real-time risk prevention and control; and third, there are systemic defects, with a lack of effective collaboration between different monitoring systems, making it impossible to form a complete closed loop for risk identification, verification, and handling. Summary of the Invention

[0004] In view of the above problems, this application provides a method, device, equipment, storage medium and program product for handling abnormal behavior. By matching the static attribute information of the target area with the generated multimodal data, the state information of the entity object set is generated, thereby classifying and handling the detected abnormal behavior. This solves the current problem of untimely identification and inaccurate handling of abnormal behavior in bank branches, and improves the real-time performance of risk prevention and control in key locations for financial institutions.

[0005] According to the first aspect of this application, an abnormal behavior handling method is provided, comprising: acquiring static attribute information of a target area and multimodal data generated by the target area, wherein the target area represents a physical area with business attributes, and the multimodal data represents data generated by a set of entity objects associated with business attributes in the target area; matching the static attribute information and the multimodal data to generate state information of the set of entity objects in the target area; detecting the state information of the set of entity objects in the target area to determine the confidence level of abnormal behavior of the set of entity objects in the target area; and handling the abnormal behavior based on the relationship between the confidence level of abnormal behavior of the set of entity objects and a preset confidence threshold.

[0006] According to an embodiment of this application, acquiring multimodal data generated in a target area includes: acquiring video stream data of the target area based on a preset video acquisition device; acquiring sensor data of the target area based on a preset IoT sensor; and spatiotemporally aligning the video stream data and sensor data to obtain multimodal data.

[0007] According to an embodiment of this application, video stream data and sensor data are spatiotemporally aligned to obtain multimodal data, including: acquiring a first time series of video stream data and a second time series of sensor data, and determining the temporal mapping relationship between the first time series and the second time series; acquiring a first spatial coordinate system of video stream data and a second spatial coordinate system of sensor data, and determining the spatial mapping relationship between the first spatial coordinate system and the second spatial coordinate system; and spatiotemporally aligning the video stream data and sensor data based on the temporal mapping relationship and the spatial mapping relationship to obtain multimodal data.

[0008] According to an embodiment of this application, the entity object set includes multiple entity objects; matching static attribute information and multimodal data to generate state information of the entity object set in the target area includes: extracting features from the multimodal data; mapping the extracted features to the static attribute information to obtain the correspondence between every two entity objects; and generating state information of the entity object set based on the correspondence between every two entity objects.

[0009] According to an embodiment of this application, the state information includes location information. Detecting the state information of a set of entity objects in a target area and determining the confidence level of abnormal behavior in the set of entity objects in the target area includes: obtaining the location information of the set of entity objects in a preset first continuous time period from the detection results; determining the displacement of the set of entity objects in the continuous time period based on the location information; and determining the confidence level of abnormal behavior in the set of entity objects in the target area based on the deviation between the displacement of the set of entity objects in the continuous time period and a preset displacement threshold.

[0010] According to an embodiment of this application, detecting the state information of a set of entity objects in a target area and determining the confidence level of abnormal behavior of the set of entity objects in the target area further includes: determining the duration for which the location information of the set of entity objects is in an unauthorized physical area from a preset second continuous time period; and determining the confidence level of abnormal behavior of the set of entity objects in the target area based on the deviation between the duration for which the location information of the set of entity objects is in an unauthorized physical area and a preset duration threshold.

[0011] According to an embodiment of this application, abnormal behavior is handled based on the relationship between the confidence level of an entity object set exhibiting abnormal behavior and a preset confidence threshold. This includes: when the confidence level is greater than or equal to the confidence threshold, determining a target confidence interval corresponding to the confidence level from a preset database. The database stores multi-level confidence intervals and multi-level handling rules. The multi-level confidence intervals include a first-level confidence interval and a second-level confidence interval. The minimum confidence level of the second-level confidence interval is the confidence threshold, and the minimum confidence level of the first-level confidence interval is the maximum confidence level of the second-level confidence interval. The multi-level handling rules are arranged in descending order of handling intensity as first-level handling rules and second-level handling rules. The multi-level confidence intervals and multi-level handling rules correspond one-to-one, and the target confidence interval is one of the multi-level confidence intervals. When the confidence level falls within the target confidence interval, retrieving the target handling rule corresponding to the target confidence interval from the database and handling the abnormal behavior according to the target handling rule.

[0012] According to an embodiment of this application, after determining the confidence level of abnormal behavior in the set of entity objects in the target area, the method further includes: if the confidence level of abnormal behavior in the set of entity objects is greater than or equal to a confidence threshold, determining an update coefficient of the confidence level based on business attributes; updating the confidence level based on the update coefficient; and handling the abnormal behavior based on the relationship between the updated confidence level and the confidence threshold.

[0013] The second aspect of this application provides an abnormal behavior handling device, comprising: a data acquisition module for acquiring static attribute information of a target area and multimodal data generated by the target area, wherein the target area represents a physical area with business attributes, and the multimodal data represents data generated by a set of entity objects associated with business attributes in the target area; a data matching module for matching the static attribute information and the multimodal data to generate state information of the set of entity objects in the target area; an anomaly detection module for detecting the state information of the set of entity objects in the target area to determine whether the set of entity objects in the target area has abnormal behavior; and an anomaly handling module for handling abnormal behavior based on the relationship between the confidence level of the abnormal behavior of the set of entity objects and a preset confidence threshold.

[0014] A third aspect of this application provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0015] A fourth aspect of this application also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0016] The fifth aspect of this application also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method. Attached Figure Description

[0017] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0018] Figure 1 The illustrations depict application scenarios of abnormal behavior handling methods, apparatus, devices, media, and program products according to embodiments of this application.

[0019] Figure 2 A flowchart illustrating an abnormal behavior handling method according to an embodiment of this application is shown schematically.

[0020] Figure 3 A schematic diagram illustrating the acquisition of multimodal data generated from a target region according to an embodiment of this application is shown;

[0021] Figure 4 This schematically illustrates a structural block diagram of an abnormal behavior handling device according to an embodiment of this application; and

[0022] Figure 5 A block diagram schematically illustrates an electronic device suitable for implementing an abnormal behavior handling method according to an embodiment of this application. Detailed Implementation

[0023] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0024] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0025] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0026] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0027] In the technical solution of this application, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0028] In scenarios involving automated decision-making using personal information, the methods, devices, and systems provided in this application all offer users corresponding entry points for choosing to agree to or reject the automated decision-making results. If the user chooses to reject, the process proceeds to the expert decision-making stage. Here, "automated decision-making" refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests, or economic, health, and credit status through computer programs, and then making a decision. Here, "expert decision-making" refers to the activity of making decisions by personnel who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.

[0029] Figure 1 The illustration shows an application scenario diagram of the abnormal behavior handling method, apparatus, device, medium, and program product according to embodiments of this application.

[0030] like Figure 1As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0031] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0032] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0033] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0034] For example, in the daily operational monitoring scenario of the voucher area in a bank branch, this area involves several key processes such as cash drawer handover, voucher allocation, and equipment operation. Operators need to strictly adhere to requirements such as two-person operation, authorized access, and fixed-position management of items. In actual operation, due to blind spots in manual inspections, ineffective linkage between video stream data and sensor data, and a lack of real-time quantitative evaluation of rule responses, problems such as delayed detection of abnormal behavior in the voucher area, reliance on human experience for handling, and inconsistent risk response standards are prone to occur. This makes it difficult to promptly stop violations and handle risk events quickly.

[0035] The abnormal behavior handling method in this application matches the static attribute information of the voucher area with the generated multimodal data to generate status information of the entity object set (such as equipment and personnel) in the voucher area. Abnormal behaviors are identified and their confidence levels are quantified. Based on the relationship between the confidence level and a preset confidence threshold, handling rules for abnormal behaviors are determined. This enables real-time monitoring and intelligent analysis of personnel operations, equipment status, and environmental safety. It effectively reduces the workload of manual monitoring, improves the real-time performance and accuracy of abnormal responses, strengthens the standardization and traceability of risk handling, and enhances the proactive security control capabilities and intelligent operation management level of bank branches in key areas.

[0036] It should be noted that the abnormal behavior handling method provided in this application embodiment can generally be executed by server 105. Correspondingly, the abnormal behavior handling device provided in this application embodiment can generally be located in server 105. The abnormal behavior handling method provided in this application embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the abnormal behavior handling device provided in this application embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0037] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0038] The following will be based on Figure 1 The described scene, through Figures 2-3 The abnormal behavior handling method according to the embodiments of this application will be described in detail.

[0039] Figure 2 A flowchart illustrating an abnormal behavior handling method according to an embodiment of this application is shown schematically.

[0040] like Figure 2 As shown, the abnormal behavior handling method in this embodiment includes operations S210 to S240.

[0041] In operation S210, static attribute information of the target area and multimodal data generated by the target area are obtained. The target area represents a physical area with business attributes, and the multimodal data represents the data generated by the set of entity objects in the target area that are associated with the business attributes.

[0042] In embodiments of this application, user consent or authorization may be obtained before acquiring multimodal data generated in the target area. For example, a request to acquire multimodal data may be sent to the user before operation S210. If the user consents or authorizes the acquisition of multimodal data, operation S210 is executed.

[0043] The target area can be a physical area whose scope and functions are clearly defined by business requirements, along with corresponding access control rules.

[0044] For example, a bank's cash transaction area handles cash deposits, withdrawals, transfers, and other cash-related transactions; a bank's vault stores store and manages financial documents; and a bank's core computer room stores and processes bank system data.

[0045] An entity object can be a person or thing existing within the target area and directly related to the business attributes of the target area; changes in its behavior or state can affect the normal operation of the business. An entity object collection can be a set of multiple entity objects existing within the target area. Static attribute information can be attribute information inherent to the target area itself, which does not change dynamically with time or the activities of the entity object collection.

[0046] For example, the set of entity objects in the cash transaction area can be tellers (personnel), customers (personnel), cash drawers (items), and safes (equipment); the set of entity objects in the bank voucher warehouse can be warehouse managers (personnel), voucher shelves (equipment), blank checks, deposit slips (items), and warehouse temperature and humidity sensors (equipment).

[0047] For example, the static attribute information of the cash handling area can be a coordinate range (such as X-axis 0m~10m, Y-axis 0m~8m, Z-axis 0m~3m) or the location parameters of fixed equipment in the area (such as the fixed coordinates of cash counters and safes).

[0048] Multimodal data can be a collection of various types of data generated by the activities of a set of entity objects associated with business attributes within a target area. Through multi-dimensional collection, a comprehensive perception of the behavior of each entity object in the set of entity objects can be achieved.

[0049] For example, the opening or closing of safes, the movement of cash drawers, and the entry and exit of personnel collected by the access control system.

[0050] In operation S220, static attribute information and multimodal data are matched to generate state information of the set of entity objects in the target area.

[0051] In the embodiments of this application, a corresponding operation entry can be provided to the user, allowing the user to choose to agree to or reject the automated decision result. That is, before matching static attribute information and multimodal data, the user can provide an instruction to agree or reject the matching through the corresponding operation entry. If the user agrees to the matching, the static attribute information and multimodal data are matched, i.e., step S220 is executed. If the user refuses to match, the expert decision-making process is initiated.

[0052] Status information can be the specific status of each entity object in the entity object collection at a specific point in time and within a specific target area, including location status, behavior status, running status, and environmental status.

[0053] For example, the teller's status information is that they are currently in the cash transaction area (location status) and are performing cash deposit and withdrawal operations (behavior status); the cash tail box's status information is that it is located 0.5 meters to the right of the cash counter (location status) and is in a closed and locked state (operational status); the status information of a blank deposit slip can be that it is stored on shelf 1 of voucher warehouse A (location status) and the warehouse temperature and humidity meet the storage requirements (environmental status).

[0054] In operation S230, the state information of the set of entity objects in the target area is detected to determine the confidence level that the set of entity objects in the target area has abnormal behavior.

[0055] Abnormal behavior can be caused by the state or behavior of entity objects in a collection of entity objects deviating from the constraints corresponding to the business attributes of the target area.

[0056] For example, unauthorized personnel enter the cash transaction area (abnormal personnel location); the cash drawer is moved beyond a preset distance threshold during non-operational hours (abnormal item displacement); tellers fail to double-check large withdrawals as required (abnormal personnel behavior); the safe is opened outside of working hours (abnormal equipment operation).

[0057] For example, abnormal behavior can be classified by subject into personnel abnormality (such as unauthorized entry, leaving the post), equipment abnormality (such as abnormal opening of safe, sensor failure), and item abnormality (such as abnormal displacement of cash box, missing voucher); and by risk level, it can be classified into general abnormality (such as customer mistakenly entering non-business area) and serious abnormality (such as unauthorized personnel touching safe).

[0058] Confidence level can be an assessment that combines relevant risk factors to determine the degree of credibility in determining the existence of the abnormal behavior.

[0059] In operation S240, abnormal behavior is handled based on the relationship between the confidence level of the entity object set having abnormal behavior and the preset confidence threshold.

[0060] Preset confidence thresholds can be threshold values ​​set in advance based on factors such as business risk tolerance, handling costs, and the impact of misjudgments. These thresholds are used to differentiate the severity of handling abnormal behavior. They can be dynamically adjusted according to specific business scenarios, with different confidence thresholds corresponding to different handling rules.

[0061] The abnormal behavior handling method based on the embodiments of this application achieves comprehensive perception of the relevant states of a set of entity objects by matching static attribute information with multimodal data, improving the accuracy of risk identification and effectively strengthening the bank's security defenses. Based on the relationship between the confidence level of abnormal behavior and the confidence threshold, it automatically executes tiered handling, reducing unnecessary business interruptions and ensuring the stability of bank operations. It forms a complete risk tracing data chain, helping banks achieve intelligent and standardized upgrades in security management and strengthening the reliability and efficiency of the overall risk prevention and control system.

[0062] In the embodiments of this application, acquiring multimodal data generated in the target area includes: acquiring video stream data of the target area based on a preset video acquisition device; acquiring sensor data of the target area based on a preset IoT sensor; and spatiotemporally aligning the video stream data and sensor data to obtain multimodal data.

[0063] The following is combined Figure 3 This application provides a detailed description of the multimodal data generated from the target region in its embodiments.

[0064] Figure 3 A schematic diagram of acquiring multimodal data generated from a target region according to an embodiment of this application is shown.

[0065] like Figure 3 As shown, two video capture devices (Video 1 and Video 2) and four IoT sensors (Sensor 1, Sensor 2, Sensor 3, and Sensor 4) were deployed in the target area. The deployment density and viewing angle of the video capture devices were determined based on the business attributes of the target area. Video 1 was deployed above the counter, using a high-altitude overhead view, matching the business attribute of clearly recording customer-teller interactions in counter operations. The captured video stream data includes customer body movements (behavioral data) and teller service status (status data). Video 2 was deployed at the entrance of the target area, using a wide-angle eye-level view, matching the business attribute of monitoring personnel flow at entrances and exits. The captured video stream data includes personnel entering and exiting actions (behavioral data) and personnel dwelling status (status data).

[0066] Sensor 1 is used to collect operational data of the counter equipment (such as the operating status of computers and queuing machines); Sensor 2 is used to collect operational data of equipment within the target area (such as the operating parameters of air conditioners and lighting); Sensor 3 is used to collect position data of customer seats (such as changes in the spatial position of customers sitting or leaving their seats); Sensor 4 is used to collect location-related data of customers' personal smart devices, which can help locate the regional distribution of a set of physical objects. By spatiotemporally aligning the collected video stream data and sensor data, multimodal data of the target area can be obtained.

[0067] For example, sensor 1 can be a network sensor, sensor 2 can be an environmental sensor, sensor 3 can be a location sensor, and sensor 4 can be a smart device sensor.

[0068] The abnormal behavior handling method based on the embodiments of this application deploys video equipment according to the business attributes of bank counters, which not only ensures the clear acquisition of the interactive behavior of the entity object set, but also avoids resource redundancy; by supplementing the data of device operation, location and other dimensions by multiple types of IoT sensors, the behavior, state, operation and location of the entity object set are realized in the whole domain.

[0069] In embodiments of this application, video stream data and sensor data are spatiotemporally aligned to obtain multimodal data, including: acquiring a first time series of video stream data and a second time series of sensor data, and determining the temporal mapping relationship between the first time series and the second time series; acquiring a first spatial coordinate system of video stream data and a second spatial coordinate system of sensor data, and determining the spatial mapping relationship between the first spatial coordinate system and the second spatial coordinate system; and spatiotemporally aligning the video stream data and sensor data based on the temporal mapping relationship and the spatial mapping relationship to obtain multimodal data.

[0070] Video capture devices can acquire images at a fixed frequency. Each frame of the image is accompanied by a timestamp generated by a hardware clock or software decoding. The timestamps corresponding to each frame of the video stream data can form a first time series. IoT sensors can acquire data at a higher acquisition frequency than the fixed frequency. Each data point records the acquisition time. The acquisition time corresponding to each data point in the sensor data can form a second time series.

[0071] For example, a precise time protocol can be used to synchronize the clocks of video acquisition devices and IoT sensors, obtaining the time mapping relationship between the first and second time series. The error can be controlled at the microsecond level, making it suitable for scenarios with high real-time requirements.

[0072] For example, when the video acquisition device and the IoT sensor cannot synchronize their clocks, time interpolation can be used for alignment. The second time series is matched to the nearest timestamp in the first time series, and a linear fit is performed on the sensor data to estimate the value of the video stream data at the corresponding moment, thus determining the time mapping relationship between the first and second time series. There may be a delay in the transmission of sensor data from acquisition to the processing unit; this delay can be compensated for by calibration or real-time measurement of the delay time and added to the timestamps.

[0073] The first spatial coordinate system can be a two-dimensional coordinate system with the top left corner of the image as the origin and pixels as the unit, used to describe the position of the target in the image; the second spatial coordinate system can be a three-dimensional spatial coordinate system measured by the sensor, such as a lidar sensor with itself as the origin, measuring the relative distance and angle of obstacles.

[0074] For example, by obtaining camera intrinsic parameters (such as focal length and principal point) and extrinsic parameters (such as rotation matrix and translation vector) through a calibration board (such as a checkerboard), a spatial mapping relationship between the first spatial coordinate system and the second spatial coordinate system can be established.

[0075] The abnormal behavior handling method based on the embodiments of this application eliminates the time drift of multimodal data through time synchronization, ensuring the correspondence between behavioral events and state changes and avoiding misjudgments caused by time sequence misalignment; it achieves coordinate unification of the first spatial coordinate system and the second spatial coordinate system through spatial mapping, accurately associating the set of entity objects in the video stream data with the set of entity objects in the sensor data, enhancing the positioning accuracy of entity objects in the set of entity objects; the fused multimodal data has temporal consistency and spatial consistency, which can cross-verify abnormal behavior and reduce the false alarm rate and false negative rate of a single data source.

[0076] In the embodiments of this application, the entity object set includes multiple entity objects; matching static attribute information and multimodal data to generate state information of the entity object set in the target area includes: extracting features from the multimodal data; mapping the extracted features to the static attribute information to obtain the correspondence between every two entity objects; and generating state information of the entity object set based on the correspondence between every two entity objects.

[0077] For example, deep learning models can be used to extract visual features of a collection of entity objects in multimodal data, including target categories (such as people), spatial locations (bounding box coordinates), and motion features (optical flow, trajectory).

[0078] For example, cosine similarity and Euclidean distance can be used to calculate the similarity between the extracted features and the static attribute information. The system will calculate the similarity between the extracted features and all entity objects in the static attribute information. For each candidate entity object, other entity objects with similarity higher than a preset similarity threshold can be associated with the candidate entity object to obtain the correspondence between each pair of entity objects.

[0079] The abnormal behavior handling method based on the embodiments of this application integrates the dynamic behavioral characteristics (i.e., extracted features) and static attribute information of entities through the state information of the entity object set generated by the mapping relationship, forming a structured state description with spatiotemporal consistency, thereby improving the accuracy of behavior analysis.

[0080] In the embodiments of this application, the state information includes location information. Detecting the state information of the entity object set in the target area and determining the confidence level of abnormal behavior of the entity object set in the target area includes: obtaining the location information of the entity object set in a preset first continuous time period from the detection results; determining the displacement of the entity object set in the continuous time period based on the location information; and determining the confidence level of abnormal behavior of the entity object set in the target area based on the deviation between the displacement of the entity object set in the continuous time period and a preset displacement threshold.

[0081] For example, geofencing can be used to delineate the physical boundaries of different areas within a target region, with each area assigned a unique identifier (such as region ID or name).

[0082] For example, an object detection algorithm can be used to obtain the location information of each entity object in a set of entity objects within a preset first continuous time period.

[0083] The abnormal behavior handling method based on the embodiments of this application detects abnormal behavior by tracking the displacement of a set of entity objects within a first continuous time period and comparing it with a preset displacement threshold. This improves the accuracy and response speed for identifying spatial violation-type abnormal behaviors. Simultaneously, by combining multi-dimensional judgment of temporal displacement, it effectively overcomes the limitations of single-point or single-frame detection and reduces the false alarm rate caused by instantaneous interference.

[0084] In the embodiments of this application, detecting the state information of the entity object set in the target area and determining the confidence level of the entity object set having abnormal behavior in the target area further includes: determining the duration of the entity object set's location information being in an unauthorized physical area from a preset second continuous time period; and determining the confidence level of the entity object set having abnormal behavior in the target area based on the deviation between the duration of the entity object set's location information being in an unauthorized physical area and a preset duration threshold.

[0085] For each entity in the entity object collection, a list of allowed areas is preset. The location information of each entity object within a preset second consecutive time period is checked to see if it is in the list of allowed areas. If the location information is not in the list of allowed areas, the duration of the entity object's location information in an unauthorized physical area is determined. The sum of the duration of each entity object's location information in an unauthorized physical area is taken as the total duration of the entity object collection's location information in an unauthorized physical area.

[0086] The abnormal behavior handling method based on the embodiments of this application effectively avoids invalid alarms caused by brief accidental entry or location drift by calculating the duration of the entity object set entering the unauthorized area in the second continuous time period.

[0087] In the embodiments of this application, abnormal behavior is handled based on the relationship between the confidence level of an entity object set exhibiting abnormal behavior and a preset confidence threshold. This includes: when the confidence level is greater than or equal to the confidence threshold, determining a target confidence interval corresponding to the confidence level from a preset database. The database stores multi-level confidence intervals and multi-level handling rules. The multi-level confidence intervals include a first-level confidence interval and a second-level confidence interval. The minimum confidence level of the second-level confidence interval is the confidence threshold, and the minimum confidence level of the first-level confidence interval is the maximum confidence level of the second-level confidence interval. The multi-level handling rules are arranged in descending order of handling intensity as first-level handling rules and second-level handling rules. The multi-level confidence intervals and multi-level handling rules correspond one-to-one, and the target confidence interval is one of the multi-level confidence intervals. When the confidence level falls within the target confidence interval, retrieving the target handling rule corresponding to the target confidence interval from the database and handling the abnormal behavior according to the target handling rule.

[0088] For example, the maximum confidence level needs to be less than or equal to 1. If the target confidence level interval is a level 1 confidence level interval, it indicates that the abnormal behavior is highly credible and requires immediate action, so a level 1 action rule is adopted. If the target confidence level interval is a level 2 confidence level interval, it indicates that the abnormal behavior is relatively credible and requires close attention, so a level 2 action rule is adopted.

[0089] For example, Level 1 handling rules include blocking devices or access control systems associated with abnormal behavior; Level 2 handling rules include generating abnormal behavior warning logs and storing them in a database or triggering audible and visual alarms and pop-up notifications. This tiered handling approach allows for full-process recording and traceable management of potential abnormal behavior, while temporary protection is provided through device linkage, controlling risks while minimizing disruption to normal business operations. The non-intrusive audible and visual alarms and pop-up notifications provide a flexible response mechanism for low-confidence scenarios, both attracting the attention of relevant personnel and prompting self-inspection, and preventing business interruptions caused by misjudgments in automated handling.

[0090] For example, a pre-set large language model can be used to analyze abnormal behaviors with confidence levels higher than a pre-defined confidence threshold. Combined with a pre-trained anomaly type knowledge base, anomaly type labels for the abnormal behaviors can be output. Based on the anomaly type, the abnormal behaviors can be broken down into logical units for personnel counting and equipment status monitoring, and mapped to the corresponding detection devices.

[0091] The abnormal behavior handling method based on the embodiments of this application establishes a correspondence between confidence thresholds and handling rules. It automatically matches handling rules according to the target confidence range in which the confidence level falls, avoiding insufficient or excessive handling. This optimizes resource allocation, reduces the interference of low-risk alarms on normal operations, and ensures that high-risk abnormal behaviors are handled promptly.

[0092] In the embodiments of this application, after determining the confidence level of the abnormal behavior of the entity object set in the target area, the method further includes: if the confidence level of the abnormal behavior of the entity object set is greater than or equal to the confidence level threshold, determining the update coefficient of the confidence level based on business attributes; updating the confidence level based on the update coefficient; and handling the abnormal behavior based on the relationship between the updated confidence level and the confidence level threshold.

[0093] For example, business attributes may include business peak and security level. The first update coefficient of confidence is determined by the business peak of the entity object set within a preset time period, and the second update coefficient of confidence is determined by the security level of the target area. The first update coefficient, the second update coefficient and the confidence are multiplied to obtain the updated confidence.

[0094] For example, feedback data can be automatically collected after each treatment, and the system can dynamically adjust and update the coefficients based on the feedback data.

[0095] The abnormal behavior handling method based on the embodiments of this application dynamically updates the confidence level by combining business attributes, enabling risk assessment to adapt to the spatiotemporal changes of business scenarios, ensuring that handling resources are prioritized for higher-risk abnormal behaviors, and improving the accuracy and timeliness of overall risk prevention and control.

[0096] Based on the above-described methods for handling abnormal behavior, this application also provides an apparatus for handling abnormal behavior. The following will be combined with... Figure 4 The device is described in detail.

[0097] Figure 4 A schematic block diagram of an abnormal behavior handling device according to an embodiment of this application is shown.

[0098] like Figure 4As shown, the abnormal behavior handling device 400 in this embodiment includes a data acquisition module 410, a data matching module 420, an anomaly detection module 430, and an anomaly handling module 440.

[0099] The data acquisition module 410 is used to acquire static attribute information of the target area and multimodal data generated by the target area. The target area represents a physical area with business attributes, and the multimodal data represents data generated by the set of entity objects associated with the business attributes in the target area. In one embodiment, the data acquisition module 410 can be used to perform the operation S210 described above, which will not be repeated here.

[0100] The data matching module 420 is used to match static attribute information and multimodal data to generate state information of the entity object set in the target area. In one embodiment, the data matching module 420 can be used to perform the operation S220 described above, which will not be repeated here.

[0101] The anomaly detection module 430 is used to detect the state information of the set of entity objects in the target area and determine the confidence level that the set of entity objects in the target area has abnormal behavior. In one embodiment, the anomaly detection module 430 can be used to perform the operation S230 described above, which will not be repeated here.

[0102] The exception handling module 440 is used to handle abnormal behavior based on the relationship between the confidence level of abnormal behavior in the entity object set and a preset confidence threshold. In one embodiment, the exception handling module 440 can be used to perform the operation S240 described above, which will not be repeated here.

[0103] According to an embodiment of this application, the data acquisition module 410 includes an acquisition unit and an alignment unit. The acquisition unit is used to acquire video stream data of a target area based on a preset video acquisition device; and to acquire sensor data of the target area based on a preset IoT sensor. The alignment unit is used to perform spatiotemporal alignment of the video stream data and the sensor data to obtain multimodal data.

[0104] According to embodiments of this application, the alignment unit includes a temporal alignment subunit, a spatial alignment subunit, and a data association subunit. The temporal alignment subunit is used to retrieve a first time series of video stream data and a second time series of sensor data, and determine the temporal mapping relationship between the first and second time series. The spatial alignment subunit is used to obtain a first spatial coordinate system of the video stream data and a second spatial coordinate system of the sensor data, and determine the spatial mapping relationship between the first and second spatial coordinate systems. The data association subunit is used to perform spatiotemporal alignment of the video stream data and sensor data based on the temporal and spatial mapping relationships to obtain multimodal data.

[0105] According to an embodiment of this application, the data matching module 420 includes a feature extraction unit, a data mapping unit, and an information generation unit. The feature extraction unit extracts features from multimodal data. The data mapping unit maps the extracted features to static attribute information to obtain the correspondence between every two entity objects. The information generation unit generates state information for the entity object set based on the correspondence between every two entity objects.

[0106] According to an embodiment of this application, the anomaly detection module 430 includes a displacement determination unit and a first confidence level determination unit. The displacement determination unit is used to obtain position information of the entity object set within a preset first continuous time period from the detection results; based on the position information, it determines the displacement of the entity object set within the continuous time period. The first confidence level determination unit is used to determine the confidence level of abnormal behavior of the entity object set in the target area based on the deviation between the displacement of the entity object set within the continuous time period and a preset displacement threshold.

[0107] According to an embodiment of this application, the anomaly detection module 430 further includes a duration determination unit and a second confidence level determination unit. The duration determination unit is used to determine the duration for which the location information of the entity object set is located in an unauthorized physical area from a preset second continuous time period. The second confidence level determination unit is used to determine the confidence level that the entity object set in the target area exhibits abnormal behavior based on the deviation between the duration for which the location information of the entity object set is located in an unauthorized physical area and a preset duration threshold.

[0108] According to an embodiment of this application, the anomaly handling module 440 further includes an interval determination unit and a behavior handling unit. The interval determination unit is used to determine a target confidence interval corresponding to a given confidence level from a preset database when the confidence level is greater than or equal to a confidence threshold. The database stores multi-level confidence intervals and multi-level handling rules. The multi-level confidence intervals include a first-level confidence interval and a second-level confidence interval. The minimum confidence level of the second-level confidence interval is the confidence threshold, and the minimum confidence level of the first-level confidence interval is the maximum confidence level of the second-level confidence interval. The multi-level handling rules are arranged in descending order of handling intensity as first-level handling rules and second-level handling rules. The multi-level confidence intervals and multi-level handling rules correspond one-to-one, and the target confidence interval is one of the multi-level confidence intervals. The behavior handling unit is used to retrieve the target handling rule corresponding to the target confidence interval from the database when the confidence level falls within the target confidence interval, and handle the abnormal behavior according to the target handling rule.

[0109] According to an embodiment of this application, the abnormal behavior handling device 400 further includes a coefficient determination module and a behavior handling module. The coefficient determination module is used to determine an update coefficient for the confidence level based on business attributes when the confidence level of abnormal behavior in the entity object set is greater than or equal to a confidence threshold. The behavior handling module is used to update the confidence level based on the update coefficient; and to handle the abnormal behavior based on the relationship between the updated confidence level and the confidence threshold.

[0110] According to embodiments of this application, any multiple modules among the data acquisition module 410, data matching module 420, anomaly detection module 430, and anomaly handling module 440 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this application, at least one of the data acquisition module 410, data matching module 420, anomaly detection module 430, and anomaly handling module 440 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPMA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the data acquisition module 410, data matching module 420, anomaly detection module 430, and anomaly handling module 440 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0111] Figure 5 A block diagram schematically illustrates an electronic device suitable for implementing an abnormal behavior handling method according to an embodiment of this application.

[0112] like Figure 5 As shown, an electronic device 500 according to an embodiment of this application includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage portion 505 into a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0113] RAM 503 stores various programs and data required for the operation of electronic device 500. Processor 501, ROM 502, and RAM 503 are interconnected via bus 504. Processor 501 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 502 and / or RAM 503. It should be noted that programs may also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in one or more memories.

[0114] According to embodiments of this application, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to a bus 504. The electronic device 500 may also include one or more of the following components connected to the input / output (I / O) interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 510 as needed so that computer programs read from it can be installed into the storage section 508 as needed.

[0115] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0116] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 502 and / or RAM 503 and / or one or more memories other than ROM 502 and RAM 503 described above.

[0117] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the abnormal behavior handling method provided in the embodiments of this application.

[0118] When the computer program is executed by the processor 501, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0119] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 509, and / or installed from a removable medium 511. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0120] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by processor 501, it performs the functions defined in the system of this application embodiment. According to embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0121] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0122] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0123] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

Claims

1. A method for handling abnormal behavior, characterized in that, include: Obtain static attribute information of the target area and multimodal data generated by the target area. The target area represents a physical area with business attributes, and the multimodal data represents data generated by the set of entity objects in the target area that are associated with the business attributes. The static attribute information and the multimodal data are matched to generate the state information of the entity object set in the target area; The state information of the set of entity objects in the target area is detected to determine the confidence level that the set of entity objects in the target area has abnormal behavior; Based on the relationship between the confidence level of the abnormal behavior of the entity object set and the preset confidence threshold, the abnormal behavior is handled.

2. The method according to claim 1, characterized in that, Acquiring multimodal data generated in the target region includes: Based on a preset video acquisition device, video stream data of the target area is acquired; Based on preset IoT sensors, sensor data of the target area is collected; The video stream data and the sensor data are spatiotemporally aligned to obtain the multimodal data.

3. The method according to claim 2, characterized in that, The video stream data and the sensor data are spatiotemporally aligned to obtain the multimodal data, including: Acquire a first time series of the video stream data and a second time series of the sensor data, and determine the time mapping relationship between the first time series and the second time series; Obtain the first spatial coordinate system of the video stream data and the second spatial coordinate system of the sensor data, and determine the spatial mapping relationship between the first spatial coordinate system and the second spatial coordinate system; Based on the time mapping relationship and the spatial mapping relationship, the video stream data and the sensor data are spatiotemporally aligned to obtain the multimodal data.

4. The method according to claim 1, characterized in that, The entity object set includes multiple entity objects; the static attribute information and the multimodal data are matched to generate state information of the entity object set in the target region, including: Feature extraction is performed on the multimodal data; The extracted features are mapped to the static attribute information to obtain the correspondence between every two entity objects; Based on the correspondence between each pair of entity objects, the state information of the entity object set is generated.

5. The method according to claim 1, characterized in that, The state information includes location information. The state information of the entity object set in the target area is detected to determine the confidence level that the entity object set in the target area exhibits abnormal behavior, including: The location information of the entity object set within a preset first continuous time period is obtained from the detection results; Based on the location information, the displacement of the set of entity objects in the continuous time period is determined; Based on the deviation between the displacement of the entity object set in the continuous time period and a preset displacement threshold, the confidence level of the existence of abnormal behavior of the entity object set in the target area is determined.

6. The method according to claim 1, characterized in that, Detecting the state information of the set of entity objects in the target region and determining the confidence level that the set of entity objects in the target region exhibits abnormal behavior also includes: The duration during which the location information of the entity object set is located in an unauthorized physical area is determined from a preset second continuous time period; Based on the deviation between the duration of the entity object set's location information in an unauthorized physical area and a preset duration threshold, the confidence level of the entity object set in the target area is determined.

7. The method according to claim 1, characterized in that, Based on the relationship between the confidence level of the abnormal behavior of the entity object set and a preset confidence threshold, the abnormal behavior is handled, including: If the confidence level is greater than or equal to the confidence level threshold, a target confidence level range corresponding to the confidence level is determined from a preset database. The database stores multi-level confidence intervals and multi-level handling rules. The multi-level confidence intervals include a first-level confidence interval and a second-level confidence interval. The minimum confidence level of the second-level confidence interval is the confidence threshold, and the minimum confidence level of the first-level confidence interval is the maximum confidence level of the second-level confidence interval. The multi-level handling rules are arranged in descending order of handling intensity as first-level handling rules and second-level handling rules. The multi-level confidence intervals and the multi-level handling rules correspond one-to-one. The target confidence interval is one of the confidence intervals in the multi-level confidence intervals. If the confidence level falls within the target confidence level range, the target handling rule corresponding to the target confidence level range is retrieved from the database, and the abnormal behavior is handled according to the target handling rule.

8. The method according to claim 1, characterized in that, After determining the confidence level that the set of entity objects in the target region exhibits anomalous behavior, the method further includes: If the confidence level of the abnormal behavior in the set of entity objects is greater than or equal to the confidence level threshold, the update coefficient of the confidence level is determined based on the business attributes. The confidence level is updated based on the update coefficients; The abnormal behavior is handled based on the relationship between the updated confidence level and the confidence threshold.

9. An abnormal behavior handling device, characterized in that, The device includes: The data acquisition module is used to acquire static attribute information of a target area and multimodal data generated by the target area. The target area represents a physical area with business attributes, and the multimodal data represents data generated by a set of entity objects in the target area that are associated with the business attributes. The data matching module is used to match the static attribute information and the multimodal data to generate state information of the set of entity objects in the target area; An anomaly detection module is used to detect the state information of the entity object set in the target area to determine whether the entity object set in the target area has exhibited abnormal behavior; and The exception handling module is used to handle the abnormal behavior based on the relationship between the confidence level of the abnormal behavior of the entity object set and a preset confidence threshold.

10. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 8.

Images