Security and protection system and method

Abstract

The invention provides a security and protection system and method, and the system comprises a multi-mode monitoring module which is used for capturing a monitoring image, audio data and environment information in a security and protection region; the feature extraction module is used for extracting individual identity features and individual behavior features in the monitoring image by using an image recognition algorithm; the authority control module is used for comparing the individual identity characteristics with an initial authority library and judging the authority level corresponding to the individual; the behavior analysis module is used for grading individual behavior characteristics of the individual according to a behavior grading library; and the behavior prediction module is used for predicting the individual behavior of the individual in the prediction time sequence when the individual is in the non-monitoring image to obtain the predicted individual behavior characteristics in the prediction time sequence. According to the method, the individual behavior characteristics can be graded based on the multi-modal monitoring data in combination with the image characteristic classification algorithm, so that the behavior grades of the individuals in the security dead angles can be divided.

Description

Technical Field

[0001] The present invention belongs to the technical field of security monitoring systems, and in particular relates to a security system and method. Background Art

[0002] With the acceleration of urbanization and the increasing severity of public security issues, the construction of security monitoring systems has gradually become an important part of modern social security management. Security monitoring systems can monitor specific areas in real time through video monitoring technology, detect and respond to various emergencies in a timely manner, and provide guarantees for maintaining public safety. However, despite the continuous development of modern monitoring technology, its application process still faces some technical challenges, one of the most prominent technical problems is "security monitoring blind spots".

[0003] Security monitoring blind spots refer to areas that cannot be effectively covered by monitoring equipment due to the layout of monitoring equipment, shooting angles or field of view limitations. These blind spots may become "safe zones" for criminals, and when emergencies may occur, the monitoring system cannot obtain relevant information in the first place, which in turn leads to delayed response in emergencies and creates major security risks. For example, in crowded areas such as commercial places, residential areas, and transportation hubs, if the monitoring system has blind spots, once theft, molestation or other crimes occur, it may have serious consequences for the victims and also make it difficult to obtain evidence afterwards.

[0004] Traditional security monitoring systems are usually monitored by fixed cameras, which have limited viewing angles and coverage, and are prone to the above-mentioned "security monitoring blind spots". In many cases, due to environmental factors, building design, line of sight obstruction, etc., the existence of these blind spots is often not solved in a timely manner. For example, in some large shopping malls or production workshops, the complex structure will cause human activities in some corners to be unable to be effectively captured by surveillance cameras, which brings huge challenges to security management. Summary of the invention

[0005] A security system and method provided by the present invention can predict the behavior of an individual when located in a security blind spot in a security area based on multimodal monitoring data, such as monitoring image data, audio data, and environmental information data. At the same time, combined with an image feature classification algorithm, the individual behavior characteristics are graded, so that the behavior level of the individual when located in the security blind spot can be divided, and corresponding alarm responses can be made according to the behavior level, thereby overcoming the above-mentioned shortcomings of the prior art.

[0006] The technical solution of the present invention to solve the above technical problems is as follows:

[0007] In a first aspect, the present invention provides a security system, comprising:

[0008] A multimodal monitoring module is used to capture monitoring images, audio data and environmental information in the security area through a multimodal monitoring device and transmit them to a storage device;

[0009] A feature extraction module is used to identify surveillance images and extract individual identity features and individual behavior features in surveillance images using image recognition algorithms;

[0010] The permission control module is used to compare the individual identity characteristics with the initial permission library and determine the permission level corresponding to the individual;

[0011] A behavior analysis module, used for grading the individual behavior characteristics of the individual according to a behavior grading library;

[0012] The behavior prediction module is used to input the individual behavior characteristics, audio data and environmental information of the individual into the prediction model according to the individual's authority level, predict the individual behavior in the prediction time series when the individual is in the non-monitored image, and obtain the predicted individual behavior characteristics in the prediction time series;

[0013] The alarm warning module is used to compare the individual behavior characteristics of the individual and the predicted individual behavior characteristics within the predicted time series with the alarm behavior, generate corresponding alarm information and execute it.

[0014] In some embodiments, the permission control module includes:

[0015] The initial permission library construction submodule is used to construct the initial permission library according to the identity information of the authorized individuals in the security monitoring area;

[0016] The initial permission judgment submodule is used to compare the individual identity features in the monitoring image with the initial permission library. If the individual identity features do not exist in the initial permission library, the individual permission is marked as a monitoring level, otherwise it is marked as an authorization level.

[0017] In some embodiments, the behavior analysis module includes:

[0018] The behavior classification library construction submodule is used to construct a behavior classification library based on the existing labeled individual behavior data set and using the image feature classification algorithm. The behavior classification library includes five behavior classifications: normal behavior, suspicious behavior, abnormal behavior, dangerous behavior, and major security threat;

[0019] The behavior classification submodule is used to compare individual behavior characteristics with the behavior classification library to determine the behavior classification to which the individual behavior characteristics belong.

[0020] In some embodiments, the behavior prediction module includes:

[0021] The prediction model building submodule is used to build a prediction model based on the multimodal hidden Markov algorithm, and train the prediction model using the existing behavior feature data, audio data, and environmental information as training sets;

[0022] The prediction output submodule is used to determine whether the individual needs to be predicted according to the individual's authority level. If the individual's authority level is the monitoring level, the individual behavior characteristics of the individual and the audio data and environmental information in the corresponding time series are input into the prediction model, and the individual behavior prediction results in the prediction time series when the individual is in a non-monitoring image are output.

[0023] In a second aspect, the present invention provides a security method, comprising the following steps:

[0024] S1: Capture surveillance images, audio data and environmental information in the security area through multimodal surveillance equipment and transmit them to the storage device;

[0025] S2: Identify surveillance images and use image recognition algorithms to extract individual identity features and individual behavior features in surveillance images;

[0026] S3: Compare the individual identity characteristics with the initial permission library and determine the permission level corresponding to the individual;

[0027] S4: classify the individual behavior characteristics of the individual according to the behavior classification library;

[0028] S5: according to the authority level of the individual, the individual behavior characteristics, audio data and environmental information of the individual are input into the prediction model, and the individual behavior of the individual in the prediction time series when the individual is in the non-monitored image is predicted to obtain the predicted individual behavior characteristics in the prediction time series;

[0029] S6: Compare the individual behavior characteristics of the individual and the predicted individual behavior characteristics within the predicted time series with the alarm behavior, generate corresponding alarm information and execute.

[0030] In some embodiments, S3 includes:

[0031] S31: constructing an initial permission library based on the identity information of authorized individuals in the security monitoring area;

[0032] S32: Compare the individual identity feature in the monitoring image with the initial permission library. If the individual identity feature does not exist in the initial permission library, mark the individual permission as a monitoring level, otherwise mark it as an authorization level.

[0033] In some embodiments, S4 includes:

[0034] S41: Based on the existing labeled individual behavior dataset and using the image feature classification algorithm, a behavior classification library is constructed. The behavior classification library includes five behavior classifications: normal behavior, suspicious behavior, abnormal behavior, dangerous behavior, and major security threat;

[0035] S42: Compare the individual behavior characteristics with the behavior classification library to determine the behavior classification to which the individual behavior characteristics belong.

[0036] In some embodiments, S5 includes:

[0037] S51: Construct a prediction model based on a multimodal hidden Markov algorithm, use the existing behavior feature data, audio data, and environmental information as training sets, and train the prediction model;

[0038] S52: According to the individual's authority level, determine whether the individual needs to perform behavior prediction. If the individual's authority level is the monitoring level, the individual behavior characteristics of the individual and the audio data and environmental information in the corresponding time series are input into the prediction model, and the individual behavior prediction results in the prediction time series when the individual is in a non-monitoring image are output.

[0039] In a third aspect, the present invention provides a computer comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the security method as described in any one of the above when executing the computer program.

[0040] In a fourth aspect, the present invention provides a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the security method as described in any one of the above.

[0041] The beneficial effects of this application are:

[0042] A security system and method provided by the present application have the following beneficial effects: 1. The present application utilizes an existing labeled individual behavior data set as a training set to construct a behavior classification library, so that the behavior classification of the individual in the monitoring image can be obtained by comparing the individual behavior characteristics in the monitoring image with the behavior classification library; 2. Furthermore, for individuals marked as monitored, when they are in a non-monitored image, that is, in a monitoring blind spot, the present application utilizes a pre-trained prediction model to predict the behavior level of the individual in the monitoring blind spot time period based on the monitoring image data of the individual before and after the monitoring blind spot time period, as well as its corresponding audio and environmental data, and issues an alarm based on the prediction, thereby solving the problem of monitoring blind spots to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1This is a schematic diagram of the security system of this application;

[0044] Figure 2 This is a flow chart of the security system for this application;

[0045] Figure 3 This is a sub-flow chart of step S3 of this application;

[0046] Figure 4 This is a sub-flow chart of step S4 of this application;

[0047] Figure 5 This is a sub-flow chart of step S5 of this application. DETAILED DESCRIPTION

[0048] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0049] In order to more clearly understand the above-mentioned purpose, features and advantages of the present application, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It is to be understood that the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. The specific embodiments described herein are merely used to explain the present application, rather than to limit the present application. Based on the described embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art belong to the scope of protection of the present application.

[0050] It should be noted that, in this document, relational terms such as “first” and “second” are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.

[0051] Combination Figure 1 That is, a schematic diagram of the security system of the present application, a security system comprising:

[0052] A multimodal monitoring module is used to capture monitoring images, audio data and environmental information in the security area through a multimodal monitoring device and transmit them to a storage device;

[0053] A feature extraction module is used to identify surveillance images and extract individual identity features and individual behavior features in surveillance images using image recognition algorithms;

[0054] The permission control module is used to compare the individual identity characteristics with the initial permission library and determine the permission level corresponding to the individual;

[0055] A behavior analysis module, used for grading the individual behavior characteristics of the individual according to a behavior grading library;

[0056] The behavior prediction module is used to input the individual behavior characteristics, audio data and environmental information of the individual into the prediction model according to the individual's authority level, predict the individual behavior in the prediction time series when the individual is in the non-monitored image, and obtain the predicted individual behavior characteristics in the prediction time series;

[0057] The alarm warning module is used to compare the individual behavior characteristics of the individual and the predicted individual behavior characteristics within the predicted time series with the alarm behavior, generate corresponding alarm information and execute it.

[0058] Specifically, the security system in this embodiment mainly includes a multimodal monitoring module, a feature extraction module, an authority control module, a behavior analysis module, a behavior prediction module, and an alarm warning module. Among them, the multimodal monitoring module is mainly used to obtain monitoring data, including but not limited to monitoring image data, audio data (sound data collected in the security area) and environmental information data (time, temperature, etc.), and the way to obtain monitoring data is mainly through the multimodal sensors and monitoring probes set in the security area.

[0059] Furthermore, after acquiring the multimodal monitoring data, the monitoring data can be transmitted to a storage device, and at the same time, the individual identity and individual behavior characteristics in the monitoring image can be identified through the feature extraction module. The identification method can adopt conventional image segmentation algorithms and facial recognition algorithms, etc., so as to obtain the individual identity characteristics and individual behavior characteristics in the monitoring image.

[0060] Furthermore, in the authority control module, the individual identity characteristics are compared with the initialization authority library. It should be noted here that the initial authority library contains individual identity information marked as authorization levels (that is, the individual's ordinary behavior in the security area is authorized). Therefore, when comparing, it is only necessary to determine whether the acquired individual identity characteristics exist in the initial authority library. When the individual identity characteristics of the individual do not exist in the initial authority library, the authority can be temporarily marked as a monitoring level (that is, the individual's behavior in the security area is monitored). Of course, for different security scenarios, there may be different security level requirements. For example, in some industrial security scenarios, different security levels are set for different security areas, and different authority levels are assigned to individuals. Therefore, the authority level of the individual in the initial authority library should be adjustable according to actual needs. This embodiment only uses the two levels of authorization authority and monitoring authority as an illustration.

[0061] Furthermore, determining the individual authority level is only the first step in security measures. The next step is to classify the behavior of all individuals in the surveillance image. The classification method can be achieved by comparing the individual behavior with a pre-trained behavior classification library to obtain the behavioral characteristic classification of each individual.

[0062] On the basis of the above modules, we also consider that there are blind spots in the security area, and the behavior of individuals in the blind spots is invisible. Therefore, for this problem, the present embodiment also proposes a behavior prediction module. The behavior prediction module first determines the authority level of the individual, and determines whether it is necessary to predict the behavior of the individual in the blind spots according to the authority level. Taking the above two types of authority levels as an example, the prediction module can only predict the behavior of individuals marked as monitoring levels in the blind spots, and the specific method of prediction is mainly through the pre-trained behavior prediction model, the individual behavior characteristics of the individual in the preset time period before and after the blind spot, the corresponding audio data and the environmental information data are input into the prediction model, and the behavior characteristics of the individual in the blind spot time period can be inferred through the forward and backward algorithm. It is worth mentioning that the above example is only the simplest case, and when the security system is used in a security scene with multiple different security classification areas, the individual needs to be monitored in which security classification areas according to the authority levels of different individuals, that is, when the authority level of the individual is not higher than the classification of the security area, then the behavior of the individual in the security area will be monitored.

[0063] Finally, the alarm warning module is also an indispensable part of the security system. The alarm warning module will also pre-set the alarm behavior classification, and determine the information that needs to be alarmed and execute it through individual behavior characteristics and predicted behavior characteristics.

[0064] In some embodiments, the permission control module includes:

[0065] The initial permission library construction submodule is used to construct the initial permission library according to the identity information of the authorized individuals in the security monitoring area;

[0066] The initial permission judgment submodule is used to compare the individual identity features in the monitoring image with the initial permission library. If the individual identity features do not exist in the initial permission library, the individual permission is marked as a monitoring level, otherwise it is marked as an authorization level.

[0067] Specifically, the permission control module in the scheme of this embodiment mainly includes an initial permission library construction submodule and an initial permission judgment submodule. Among them, the initial permission library construction submodule is used to construct an initial permission library according to the authorized individual identity information in the security monitoring area, and the above-mentioned authorized individual identity information includes but is not limited to facial collection, fingerprint collection, personal identity information, etc. Similarly, for the initial permission judgment submodule, as described above for the permission control module, the initial permission library contains individual identity information marked as the authorization level (that is, the ordinary behavior of the individual in the security area is authorized), so when comparing, it is only necessary to determine whether the acquired individual identity characteristics exist in the initial permission library, and when the individual identity characteristics of the individual do not exist in the initial permission library, the permission can be temporarily marked as the monitoring level (that is, the behavior of the individual in the security area is monitored). Of course, for different security scenarios, there may be different security level requirements. For example, in some industrial security scenarios, different security levels will be set for different security areas, and different permission levels will be given to individuals. Therefore, the permission level of individuals in the initial permission library should be adjustable according to actual needs.

[0068] In some embodiments, the behavior analysis module includes:

[0069] The behavior classification library construction submodule is used to construct a behavior classification library based on the existing labeled individual behavior data set and using the image feature classification algorithm. The behavior classification library includes five behavior classifications: normal behavior, suspicious behavior, abnormal behavior, dangerous behavior, and major security threat;

[0070] The behavior classification submodule is used to compare individual behavior characteristics with the behavior classification library to determine the behavior classification to which the individual behavior characteristics belong.

[0071] Specifically, the behavior analysis module in the embodiment includes a behavior classification library construction submodule and a behavior classification submodule. Among them, the behavior classification library construction submodule is used to construct a behavior classification library based on an existing labeled individual behavior data set (the data set can be obtained through big data or historical monitoring data, and the data set should contain individual behaviors and corresponding behavior labels), and use image feature classification algorithms, such as the commonly used CNN convolutional neural network, to construct a behavior classification library. The behavior classification library includes five behavior classifications: normal behavior, suspicious behavior, abnormal behavior, dangerous behavior, and major security threats. The above five behaviors can be interpreted as follows: Normal behavior refers to walking: people walk normally in a designated area without abnormal stops; talking: normal conversations between people, without showing tension or abnormal behavior; work activities: employees perform daily work in the office, use computers, make phone calls, etc. Suspicious behavior refers to wandering: someone moves in a specific area for a long time, seemingly observing the surrounding environment, but without obvious purpose; looking around: individuals frequently check the surrounding environment, showing alertness, and may be looking for monitoring blind spots; fast walking: individuals walk quickly, but there is no obvious emergency in the environment. Abnormal behavior refers to sudden escape: the individual seems to be fleeing the scene, which may be due to some dangerous or illegal behavior; quarrel or fight: a heated quarrel between two or more people, or a physical conflict; entering an unauthorized area: an individual enters a restricted or prohibited area without a reasonable explanation. Dangerous behavior refers to armed behavior: someone holds a weapon such as a knife or gun, which obviously threatens others; crowd impact: in a crowded place, an individual pushes or tries to break through the crowd, which may cause chaos or trampling; self-harm or harm to others: an individual shows behavior of self-harm or harming others, which obviously endangers the safety of others. Major security threats refer to robbery: an individual commits robbery or theft in a specific place with clear goals and malice; offensive behavior: violent attacks on others, which may cause casualties; arson or explosion: an individual discovers suspicious activities in a public place, which may involve arson or making explosives. For the above five behavior classifications, normal behavior represents daily activities, suspicious behavior requires attention, abnormal behavior may require an alarm, dangerous behavior is a direct threat, and major security threats require immediate response. After determining the behavior classification library, it is only necessary to compare the individual behavior characteristics with the behavior library to obtain the individual's behavior classification, so as to make a corresponding response judgment.

[0072] In some embodiments, the behavior prediction module includes:

[0073] The prediction model building submodule is used to build a prediction model based on the multimodal hidden Markov algorithm, and train the prediction model using the existing behavior feature data, audio data, and environmental information as training sets;

[0074] The prediction output sub-module is used to determine whether an individual needs to have their behavior predicted based on the individual's permission level. If the individual's permission level is the monitoring level, the individual's behavior characteristics, the audio data and environmental information in the corresponding time series are input into the prediction model, and the individual behavior prediction result within the predicted time series when the individual is in a non-monitoring image is output.

[0075] Specifically, in the solution of this embodiment, the behavior prediction module mainly includes a prediction model construction sub-module and a prediction output sub-module. Among them, the prediction model construction sub-module mainly constructs a prediction model through a multi-modal hidden Markov algorithm. The main prediction idea of this prediction model is that although the behavior of an individual in a blind spot of monitoring is not visible, it can actually be inferred through other observation information. For example, the behavior of an individual in a blind spot of monitoring can be speculated through audio information (such as arguing sounds, conversation sounds or other environmental audio data), and for another example, the behavior of an individual in a blind spot of monitoring can be speculated based on environmental information (such as temperature, time period, weather, etc.). Therefore, this model can use the existing behavior characteristic data, audio data and environmental information as the training set (similarly, this training set can be composed of big data or historical data), and the specific composition of the model can be expressed as:

[0076] Define the state set: S = s 1 , s 2 ... s 5 , where s 1 ~s 5 respectively represent the five behavior classifications of normal behavior, suspicious behavior, abnormal behavior, dangerous behavior and major safety threats.

[0077] Define the observation set: where m represents the observation mode, that is, in the solution of this embodiment, it includes audio data and environmental information.

[0078] Define the initial state probability distribution: π = π i = [P(s i )], where P(s i ) represents the probability of entering the s i state at the initial time.

[0079] Define the state probability transition matrix: A = [a ij = [P(s j 丨s i )], where a ij represents the probability of transitioning from the state s i to s j .

[0080] Define the observation probability matrix: where represents that under the m observation mode, at time t, from the state si Generate observations The probability of Represents the observation at time t in the mth observation mode.

[0081] After completing the above definition, prediction is performed through the forward-backward algorithm:

[0082] Forward algorithm: where α t (i) means that the state is s at time t i The forward probability, α ji Indicates that from state s j Transfer to state s i The probability of Indicates that in state s i Generate observations probability.

[0083] Backward Algorithm: where β t (i) means that the state is s at time t i The backward probability of Indicates that in state s j Generate observations probability.

[0084] Then, calculate the update of transition probability: where ξ t (i,j) indicates that the state is s at time t i And at time t+1 the state is s j The joint probability of t (i) indicates that the state is s at time t i probability.

[0085] And, calculate the update of the observation probability: in Indicates that in state s i Can be observed characteristics.

[0086] At this point, the construction of the prediction model is completed. Next, the existing behavior feature data, audio data, and environmental information can be used as a training set (similarly, the training set can be composed of big data or historical data) to train the above model until the model converges. Then, we can input the collected individual behavior features, audio data, and environmental information into the prediction model to output the probability of the individual behavior features of the individual during the monitoring blind spot time, and then compare it with the behavior classification library to obtain the individual behavior classification of the individual during the monitoring blind spot time, and then make corresponding warning measures.

[0087] The second aspect of the present invention also provides a security method, comprising the following steps:

[0088] S1: Capture surveillance images, audio data and environmental information in the security area through multimodal surveillance equipment and transmit them to the storage device;

[0089] S2: Identify surveillance images and use image recognition algorithms to extract individual identity features and individual behavior features in surveillance images;

[0090] S3: Compare the individual identity characteristics with the initial permission library and determine the permission level corresponding to the individual;

[0091] S4: classify the individual behavior characteristics of the individual according to the behavior classification library;

[0092] S5: according to the authority level of the individual, the individual behavior characteristics, audio data and environmental information of the individual are input into the prediction model, and the individual behavior of the individual in the prediction time series when the individual is in the non-monitored image is predicted to obtain the predicted individual behavior characteristics in the prediction time series;

[0093] S6: Compare the individual behavior characteristics of the individual and the predicted individual behavior characteristics within the predicted time series with the alarm behavior, generate corresponding alarm information and execute.

[0094] In some embodiments, the above S3 includes:

[0095] S31: constructing an initial permission library based on the identity information of authorized individuals in the security monitoring area;

[0096] S32: Compare the individual identity feature in the monitoring image with the initial permission library. If the individual identity feature does not exist in the initial permission library, mark the individual permission as a monitoring level, otherwise mark it as an authorization level.

[0097] In some embodiments, the above S4 includes:

[0098] S41: Based on the existing labeled individual behavior dataset and using the image feature classification algorithm, a behavior classification library is constructed. The behavior classification library includes five behavior classifications: normal behavior, suspicious behavior, abnormal behavior, dangerous behavior, and major security threat;

[0099] S42: Compare the individual behavior characteristics with the behavior classification library to determine the behavior classification to which the individual behavior characteristics belong.

[0100] In some embodiments, the above S5 includes:

[0101] S51: Construct a prediction model based on a multimodal hidden Markov algorithm, use the existing behavior feature data, audio data, and environmental information as training sets, and train the prediction model;

[0102] S52: According to the individual's authority level, determine whether the individual needs to perform behavior prediction. If the individual's authority level is the monitoring level, the individual behavior characteristics of the individual and the audio data and environmental information in the corresponding time series are input into the prediction model, and the individual behavior prediction results in the prediction time series when the individual is in a non-monitoring image are output.

[0103] The third aspect of the present invention also provides a computer, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the security method as described in any one of the above when executing the computer program.

[0104] The fourth aspect of the present invention further provides a readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the security method as described in any one of the above.

[0105] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0106] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0107] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0108] In the embodiments provided in the present disclosure, it should be understood that the disclosed apparatus / computer equipment and methods can be implemented in other ways. For example, the apparatus / computer equipment embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. Multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection of the apparatus or unit, which may be electrical, mechanical or other forms.

[0109] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0110] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0111] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present disclosure implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.

[0112] Those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features from different embodiments is meant to be within the scope of the present application and to form different embodiments.

[0113] Those skilled in the art will appreciate that the description of each embodiment has its own emphasis, and for parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0114] Although the embodiments of the present application are described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present application, and such modifications and variations all fall within the scope defined by the attached claims. The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or substitutions within the technical scope disclosed by the present invention, and these modifications or substitutions should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

[0115] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A security system, characterized in that: include: A multimodal monitoring module is used to capture monitoring images, audio data and environmental information in the security area through a multimodal monitoring device and transmit them to a storage device; A feature extraction module is used to identify surveillance images and extract individual identity features and individual behavior features in surveillance images using image recognition algorithms; The permission control module is used to compare the individual identity characteristics with the initial permission library and determine the permission level corresponding to the individual; A behavior analysis module, used for grading the individual behavior characteristics of the individual according to the behavior grading library; The behavior prediction module is used to input the individual behavior characteristics, audio data and environmental information of the individual into the prediction model according to the individual's authority level, predict the individual behavior in the prediction time series when the individual is in the non-monitored image, and obtain the predicted individual behavior characteristics in the prediction time series; The alarm warning module is used to compare the individual behavior characteristics of the individual and the predicted individual behavior characteristics within the predicted time series with the alarm behavior, generate corresponding alarm information and execute it.

2. A security system according to claim 1, characterized in that: The authority control module includes: The initial permission library construction submodule is used to construct the initial permission library according to the identity information of the authorized individuals in the security monitoring area; The initial permission judgment submodule is used to compare the individual identity features in the monitoring image with the initial permission library. If the individual identity features do not exist in the initial permission library, the individual permission is marked as a monitoring level, otherwise it is marked as an authorization level.

3. A security system according to claim 2, characterized in that: The behavior analysis module includes: The behavior classification library construction submodule is used to construct a behavior classification library based on the existing labeled individual behavior data set and using the image feature classification algorithm. The behavior classification library includes five behavior classifications: normal behavior, suspicious behavior, abnormal behavior, dangerous behavior, and major security threat; The behavior classification submodule is used to compare individual behavior characteristics with the behavior classification library to determine the behavior classification to which the individual behavior characteristics belong.

4. A security system according to claim 3, characterized in that: The behavior prediction module includes: The prediction model building submodule is used to build a prediction model based on the multimodal hidden Markov algorithm, and train the prediction model using the existing behavior feature data, audio data, and environmental information as training sets; The prediction output submodule is used to determine whether the individual needs to be predicted according to the individual's authority level. If the individual's authority level is the monitoring level, the individual behavior characteristics of the individual and the audio data and environmental information in the corresponding time series are input into the prediction model, and the individual behavior prediction results in the prediction time series when the individual is in a non-monitoring image are output.

5. A security method, characterized in that: The following steps are involved: S1: Capture surveillance images, audio data and environmental information in the security area through multimodal surveillance equipment and transmit them to the storage device; S2: Identify surveillance images and use image recognition algorithms to extract individual identity features and individual behavior features in surveillance images; S3: Compare the individual identity characteristics with the initial permission library and determine the permission level corresponding to the individual; S4: classify the individual behavior characteristics of the individual according to the behavior classification library; S5: according to the authority level of the individual, the individual behavior characteristics, audio data and environmental information of the individual are input into the prediction model, and the individual behavior of the individual in the prediction time series when the individual is in the non-monitored image is predicted to obtain the predicted individual behavior characteristics in the prediction time series; S6: Compare the individual behavior characteristics of the individual and the predicted individual behavior characteristics within the predicted time series with the alarm behavior, generate corresponding alarm information and execute.

6. A security method according to claim 5, characterized in that: The S3 includes: S31: constructing an initial permission library based on the identity information of authorized individuals in the security monitoring area; S32: Compare the individual identity feature in the monitoring image with the initial permission library. If the individual identity feature does not exist in the initial permission library, mark the individual permission as a monitoring level, otherwise mark it as an authorization level.

7. A security method according to claim 6, characterized in that: The S4 includes: S41: Based on the existing labeled individual behavior dataset and using the image feature classification algorithm, a behavior classification library is constructed. The behavior classification library includes five behavior classifications: normal behavior, suspicious behavior, abnormal behavior, dangerous behavior, and major security threat; S42: Compare the individual behavior characteristics with the behavior classification library to determine the behavior classification to which the individual behavior characteristics belong.

8. A security method according to claim 7, characterized in that: The S5 includes: S51: Construct a prediction model based on a multimodal hidden Markov algorithm, use the existing behavior feature data, audio data, and environmental information as training sets, and train the prediction model; S52: According to the individual's authority level, determine whether the individual needs to perform behavior prediction. If the individual's authority level is the monitoring level, the individual behavior characteristics of the individual and the audio data and environmental information in the corresponding time series are input into the prediction model, and the individual behavior prediction results in the prediction time series when the individual is in a non-monitoring image are output.

9. A computer comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the security method according to any one of claims 5 to 8 when executing the computer program.

10. A readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the security method according to any one of claims 5 to 8.

Images