Panoramic portrait intelligent monitoring system for personnel behavior in micro spaces

Abstract

The present invention discloses a panoramic portrait intelligent supervision system for personnel behavior in micro-spaces, including ubiquitous perception module, intelligent learning and training module, image deconstruction module, audio deconstruction module, panoramic three-dimensional management module, comprehensive portrait module, micro-space early warning and supervision module. Based on real-time ubiquitous perception and big data intelligent analysis and calculation, 24-hour all-weather duty for the supervision environment and personnel is realized. Through computer automation and intelligent early warning disposal, the workload of police officers can be greatly reduced, human resource investment can be reduced, and management efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of special personnel supervision using compulsory measures, and in particular to a panoramic portrait intelligent supervision system for personnel behavior in micro spaces. Background Art

[0002] Compulsory measures sites, represented by detention centers, jails, prisons, and detention centers, face various special groups of people suspected of crimes or crimes. Due to the complexity of the sources of such groups, the diversity of hierarchical structures, and the particularity of their identities, their supervision environment has specific characteristics. In the micro-space scenes of centralized management, it is easy to cause emotional out-of-control due to the failure of various communication activities, resulting in more conflicts and violations. At present, the management of these scenes is mainly carried out through real-time monitoring by deploying technical means such as cameras and voice acquisition devices, combined with the daily management of the supervising police, through the "human defense + technical defense" method, to achieve normal supervision of these scenes, but this method has many defects:

[0003] First, the emotional control of people subject to compulsory measures can only be carried out by relying on the personal experience of the supervising police officers. As each supervising police officer is responsible for managing more than 50 people, it is difficult to timely control and effectively relieve the emotions of all personnel due to the differences and limitations of personal energy and experience. Even with the help of video surveillance, it only solves the problem of information collection channels and cannot provide timely emotional control capabilities.

[0004] Second, although many management norms have been formulated in daily management work and the control capabilities have been improved through the "collective self-governance of personnel + dynamic patrols by police officers", these management norms are often superficial due to the lack of intelligent supervision and early warning analysis methods. There are also a large number of blind spots in police patrols, and it is impossible to achieve all-weather, no-dead-angle, and precise control of personnel behavior in micro-space scenes. It also consumes a large amount of supervisory police force and has low management efficiency.

[0005] With the development of modern information technologies such as cloud computing and big data, the artificial intelligence technology system built by combining big data with computer algorithms has gradually moved from experimental research to the application field. Therefore, the present invention makes full use of technical means such as big data, the Internet of Things, spatiotemporal information, and artificial intelligence to propose a panoramic portrait intelligent supervision system for personnel behavior in micro-spaces, aiming to improve the efficiency and level of compulsory measures site management by designing a panoramic and intelligent supervision of emotional changes and behavioral activities for special personnel in micro-space scenes. Summary of the invention

[0006] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a panoramic portrait intelligent supervision system for personnel behavior in micro-spaces, so as to effectively solve the problem of real-time and all-round supervision of personnel and their behavior in micro-space environments of various compulsory measures places, improve supervision efficiency, reduce management costs, and improve service levels.

[0007] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0008] A panoramic portrait intelligent supervision system for personnel behavior in micro-spaces, the key of which is: including ubiquitous perception module, intelligent learning and training module, image deconstruction module, audio deconstruction module, panoramic three-dimensional management module, comprehensive portrait module, micro-space early warning and supervision module, among which:

[0009] The ubiquitous perception module is used to comprehensively perceive and collect information about the environmental status of the micro-space of the supervision site, and access the external information of the business management system related to the supervised personnel, so as to realize the real-time, comprehensive perception, collection and storage management of the supervised personnel's behavior activities and spatial environment conditions;

[0010] The intelligent learning and training module is used to classify, parse, identify and extract the audio and video information collected by the ubiquitous perception module, analyze and summarize the emotional characteristics of personnel through the trained personnel images, voices, and behavioral characteristics, and manage and verify the learning and training results; the image deconstruction module is used to collect the audio and video information collected by the ubiquitous perception module,

[0011] Based on the video image information of the set, the facial features, behavioral dynamics and various items in the supervision place are automatically extracted and recognized in real time based on the training library support provided by the intelligent learning and training module, and the extracted and recognized elements are classified and deconstructed according to the image classification in the training library;

[0012] The audio deconstruction module automatically extracts and recognizes the dynamic voice information of personnel in real time based on the training library support provided by the above-mentioned intelligent learning and training module for the audio information collected by the ubiquitous perception module, and selects, matches, classifies and deconstructs the audio information according to the voice classification in the training library and the pre-collected vocal membrane information of personnel, and pairs the obtained data with the supervised personnel;

[0013] The panoramic three-dimensional management module is used to perform real-life three-dimensional modeling and management of the micro-space of the supervision site based on the three-dimensional GIS technology;

[0014] The comprehensive portrait module is used to integrate the information processed by the image deconstruction module and the audio deconstruction module based on the basic information of the supervised persons and on a person-by-person basis, and to carry out personnel behavior analysis and emotion analysis based on the training library provided by the intelligent learning and training module, so as to realize a panoramic real-time portrait of the individual personnel, and at the same time to carry out group portraits based on the micro-space scene of the cell as a unit, so as to reflect the all-round status of individuals, groups, and micro-space scenes through comprehensive portraits;

[0015] The micro-space early warning and supervision module is used to rely on the data of the intelligent learning and training module and the comprehensive portrait module, and based on real-time dynamic personnel portraits and environmental monitoring information, conduct real-time monitoring and analysis of the individual and group situations in the micro-space, and conduct classified and graded early warnings.

[0016] Furthermore, the ubiquitous perception module includes an image perception submodule, an audio perception submodule, an environment perception submodule, and a data aggregation and management submodule, wherein:

[0017] The image perception submodule is used to monitor the entire scene of the micro space of the supervision site without blind spots, and store and manage the obtained video image data;

[0018] The audio perception submodule is used to collect highly restored voice information of personnel in the micro-space of the supervision site, and store and manage the collected audio information; the environmental perception submodule is used to collect temperature and humidity data of the micro-space of the supervision site, and store and manage the collected environmental perception information;

[0019] The data aggregation and management submodule is used to collect, integrate, convert channels and store various types of data provided by the image perception submodule, audio perception submodule and environmental perception submodule, and connect with external related business management systems to collect personnel and business information from external sources.

[0020] Furthermore, the intelligent learning and training module includes a feature dictionary classification submodule, an artificially extracted feature classification submodule, a feature deep learning classification submodule, a comprehensive analysis submodule, and a feature library management submodule, wherein:

[0021] The feature dictionary classification submodule is used to use a pre-established dictionary library to divide and classify image information and audio information, and to achieve classification and extraction of image and audio features of human expressions, behaviors, voices, and objects in microspaces through multi-granularity combined calculations;

[0022] The manual feature extraction classification submodule is used to select certain person images, object images, and audio information as analysis samples, manually extract facial expressions, behaviors, voices, and object features in the analysis samples, and use an algorithm model to automatically classify and extract implicit features of other images and audio information other than the analysis samples;

[0023] The feature deep learning classification submodule is used to establish a massive feature database of expressions, behaviors, voices, and objects through big data collection and induction, and then automatically classify and extract implicit features of image and audio information;

[0024] The comprehensive analysis submodule is used to analyze and calculate the classification features of the same object extracted by the feature dictionary classification submodule, the manual extraction feature classification submodule, and the feature deep learning classification submodule using a multi-factor weighted calculation method, and obtain a comprehensive analysis result on the basis of retaining the three types of extraction results, and then output it to the feature library management submodule;

[0025] The feature library management submodule is used to implement the management function of the classification feature data output by the comprehensive analysis submodule.

[0026] Furthermore, the image deconstruction module includes a person expression analysis submodule, a person behavior analysis submodule, and an object analysis submodule, wherein:

[0027] The personnel expression analysis submodule is used to dynamically analyze the expressions of personnel in the video perception images collected by the ubiquitous perception module by using the facial expression feature classification extraction feature library provided by the intelligent learning and training module, and output the analysis results to the comprehensive portrait module;

[0028] The personnel behavior analysis submodule is used to analyze the personnel behavior dynamics in the video perception image collected by the ubiquitous perception module by using the behavior dynamic feature classification extraction feature library provided by the intelligent learning and training module, and output the analysis results to the comprehensive portrait module;

[0029] The object analysis submodule is used to use the object feature classification provided by the intelligent learning and training module to extract the feature library, analyze and track the objects in the video perception image collected by the ubiquitous perception module, focus on accurately extracting and dynamically marking and tracking prohibited items, and output the analysis results to the micro-space early warning and supervision module.

[0030] Furthermore, the personnel expression analysis submodule also divides the expression features through two levels. One is to classify the expressions into two categories, positive emotion and negative emotion, according to the emotion polarity classification. The other is to classify the expressions into happy, joyful, nervous, fearful, resentful, indifferent, angry, and sad according to multiple emotion categories. The personnel behavior analysis submodule also divides the behavior features through two levels. One is to classify the behaviors into two categories, positive emotion and negative emotion, according to the emotion polarity classification. The other is to classify the behaviors into happy, joyful, nervous, fearful, resentful, indifferent, angry, and sad according to multiple emotion categories.

[0031] Furthermore, the audio deconstruction module includes an audio selection submodule, a vocal membrane matching submodule, and a speech analysis submodule, wherein:

[0032] The audio selection submodule is used to select the voice and other audio information emitted by the personnel from the audio information collected by the ubiquitous perception module by using the audio feature library provided by the intelligent learning and training module;

[0033] The vocal membrane matching submodule is used to access the vocal membrane information of the personnel extracted by the ubiquitous perception module, pair and analyze the voice information of the personnel selected in the audio selection submodule with the vocal membrane information of the personnel, and bind the perceived voice with the personnel;

[0034] The speech analysis submodule is used to extract a feature library based on the audio feature classification provided by the intelligent learning and training module, analyze the voice information of the personnel in the audio information, and output the analysis results to the comprehensive portrait module after classifying the voice features of the personnel.

[0035] Furthermore, the panoramic 3D management module includes a GIS engine submodule, a micro space model management submodule, a micro space grid management submodule, a micro space scene management submodule, and a panoramic 3D service interface submodule, wherein:

[0036] The GIS engine submodule is used to provide management, analysis and calculation of spatial data; the micro space model management submodule is based on the basic functions of the GIS engine submodule to realize the addition, deletion, modification and encoding of the three-dimensional model in the micro space;

[0037] The micro-space grid management submodule divides the micro-space plane into grids based on the basic functions of the GIS engine submodule, and manages the grid units formed by the division;

[0038] The micro-space scene management submodule is based on the basic functions of the GIS engine submodule to add, delete, and encode GIS space objects in the micro-space except for the three-dimensional model objects;

[0039] The three-dimensional service interface submodule is used to open the basic functions, micro-space model data, micro-space grid data, and micro-space scene data of the GIS engine submodule in an interface componentized manner in a universal format interface manner.

[0040] Furthermore, the comprehensive portrait module includes a personnel basic information management submodule, a behavior and emotion analysis submodule, a personnel portrait submodule, and a micro-space portrait submodule, wherein:

[0041] The personnel basic information management submodule is used to uniformly manage the basic information of supervised personnel by aggregating personnel information from an external business management system;

[0042] The behavior and emotion analysis submodule is used to integrate the information on the expression, behavior, and voice features of the personnel analyzed by the received image deconstruction module and the audio deconstruction module, and to perform a comprehensive analysis based on the micro-space environment status and basic personnel information to obtain the behavior and emotion analysis results of the supervised personnel as individuals;

[0043] The personnel portrait submodule is used to use the results of behavior and emotion analysis to conduct portrait management of supervised personnel; the micro-space portrait submodule is used to take a single micro-space as a unit and the group of supervised personnel in each micro-space as a whole to portrait the group's behavior, emotion and stability changes.

[0044] Furthermore, the personnel portrait submodule conducts portrait management on the supervised personnel in two ways: static portrait and dynamic portrait. The static portrait is a portrayal of the personality, emotions, and behavioral characteristics of the personnel based on the basic information of the personnel and the results of long-term dynamic monitoring and analysis of the personnel, while the dynamic portrait is a portrayal of the real-time emotions and behavioral characteristics of the personnel based on the real-time dynamic information and analysis results of ubiquitous perception.

[0045] Furthermore, the micro-space warning and supervision module includes a personnel abnormal warning submodule, a group abnormal warning submodule, a prohibited item dynamic monitoring submodule, and a comprehensive supervision submodule, wherein:

[0046] The personnel abnormality warning submodule is used to classify the personnel abnormality into four levels of green, orange, yellow and red through threshold adjustment according to the personnel dynamic portrait information of the comprehensive portrait module, which respectively represent the four state intervals of stable, relatively unstable, easily excited and excited;

[0047] The group abnormality warning submodule is used to monitor and analyze the group abnormality based on the dynamic information of the comprehensive portrait module and the personnel abnormality warning submodule, taking a single cell microspace as a unit, and divides the group abnormality into four levels of green, orange, yellow and red through threshold adjustment, which respectively represent the four state intervals of group stability, group instability, group prone to radicalism and group radicalism;

[0048] The prohibited items dynamic detection submodule is used to dynamically detect and track prohibited items that may exist in the micro space;

[0049] The comprehensive supervision submodule uses the three-dimensional GIS scene as a carrier to carry out comprehensive and visual supervision of people and groups in the micro space, and connects with the external business management system to output relevant analysis results to the external business system.

[0050] The remarkable effects of the present invention are:

[0051] First, make full use of big data and artificial intelligence technologies to change the traditional supervision model that is mainly based on the personal experience of police officers. Through information and intelligent means, personal experience is integrated into collective experience to provide more scientific, accurate and comprehensive supervision support;

[0052] Second, based on the combination of GIS spatial analysis and multiple deep learning algorithms, the traditional qualitative analysis is refined into a quantitative analysis model, which can provide more accurate judgment basis for both macro and micro analysis;

[0053] 3. Based on real-time ubiquitous perception and big data intelligent analysis and calculation, 24-hour all-weather monitoring of the supervision environment and personnel can be achieved. Through computer automation and intelligent early warning and disposal, the workload of police officers can be greatly reduced, human resource investment can be reduced, and management efficiency can be improved;

[0054] 4. Based on the accumulation of long-term perception and monitoring data as well as artificial intelligence computing and analysis data, we can continuously improve the ability to control and analyze personnel behaviors and emotions, as well as predict and analyze them, and enhance the ability to prevent emergencies, thus transforming the traditional "post-event handling" into a "pre-event prevention" working model. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] FIG1 is a block diagram of the system structure of the present invention;

[0056] Figure 2 is a schematic diagram of the structure of the ubiquitous perception module;

[0057] FIG3 is a schematic diagram of the structure of the intelligent learning and training module;

[0058] FIG4 is a feature extraction flow chart of the feature deep learning classification submodule;

[0059] FIG5 is a schematic diagram of the structure of the image deconstruction module;

[0060] FIG6 is a schematic diagram of the structure of the audio deconstruction module;

[0061] FIG7 is a schematic diagram of the structure of a panoramic three-dimensional management module;

[0062] FIG8 is a schematic diagram of the structure of the comprehensive portrait module;

[0063] Figure 9 is a schematic diagram of the structure of the micro-space early warning and supervision module. DETAILED DESCRIPTION

[0064] The specific implementation manner and working principle of the present invention are further described in detail below with reference to the accompanying drawings.

[0065] As shown in Figure 1, a panoramic portrait intelligent supervision system for personnel behavior in micro-spaces is composed of ubiquitous perception module, intelligent learning and training module, image deconstruction module, audio deconstruction module, panoramic three-dimensional management module, comprehensive portrait module, and micro-space early warning and supervision module, among which:

[0066] The ubiquitous perception module mainly uses the Internet of Things technology to comprehensively perceive and collect information on the video, audio, temperature and humidity and other environmental conditions of the micro-space of the supervision site, and access the external information of the business management system related to the supervised personnel to achieve real-time, comprehensive perception, collection and storage management of personnel behavior activities and spatial environment conditions;

[0067] The external information refers to information in other relevant personnel, case management and business systems, such as basic personal information and family situation information in the household registration management system, basic personal information and accommodation information in the floating population management system, case situation information in the case management system and other types of information.

[0068] The intelligent learning training module uses three methods, namely, feature dictionary classification, manually extracted feature classification, and feature deep learning classification, to classify, parse, and identify the audio and video information collected by the ubiquitous perception module. It analyzes and summarizes the emotional characteristics of personnel through the trained image, voice, and behavior characteristics of personnel, and manages and verifies the learning and training results, forming a cyclical machine automatic learning system to support the intelligent operation of other functional modules of the system.

[0069] The image deconstruction module automatically extracts and identifies facial features, behavioral dynamics, and various items in the supervision site in real time based on the video image information collected by the camera in the micro space, based on the training library support provided by the intelligent learning and training module, and classifies and deconstructs the extracted and identified elements according to the image classification in the training library, deconstructing the continuous dynamic unstructured video data into discrete, standardized structured data to support analysis, early warning, and supervision work;

[0070] In this example, the image deconstruction module performs real-time automatic extraction and identification of facial features, behavioral dynamics, and various items in the supervision site: according to the facial features, behavioral dynamics, and items of the personnel, through mature image recognition algorithms, and based on the classified deconstruction of image data, various video images are dynamically extracted to establish an image library, in which the facial and behavioral dynamic image library is stored in units of individuals, and the object image library is stored in units of independent micro-spaces. In the initial state, the system will extract, identify, and store all image information that has not been collected and extracted, but with the enrichment and improvement of the corresponding classified deconstruction image data, only images with feature changes exceeding a certain threshold (such as 30%) are stored after subsequent dynamic extraction and identification. In the above-mentioned image recognition algorithm, there are mature big data recognition methods for the recognition of facial features and behavioral dynamics of personnel, which can be directly used. For the recognition of object images, due to the particularity of the compulsory measures site, various types of objects are relatively fixed and foreseeable, so it is possible to first establish a feature image dictionary library of the object, build a full-volume image library of the possible object objects, and then extract and identify the object objects through pixel matching.

[0071] The method of classifying and deconstructing the extracted and identified elements according to the image classification in the training library is: it refers to the process of classifying and analyzing the extracted and identified image data to construct structured data. Since the extraction and identification process needs to refer to the existing classified deconstruction data, if there is corresponding classified deconstruction data in the extraction and identification process, the classification label and coding label are directly added to the extracted and identified image according to the corresponding classification results, and a classified deconstruction index table is established. For image data that has no reference in the classified deconstruction data, the computer automatically matches and calculates according to the law of feature points of image recognition, and gives the corresponding pre-judgment results. Then the supervisory cadres make the final classification based on the pre-judgment results in combination with their personal experience. For example, in the recognition of facial features, the computer extracts facial feature points based on the extracted facial image, and then classifies the facial image into expressions such as smiling, crying, indifference, excitement, and anger based on the distance ratio between multiple feature points. The supervisory cadres then confirm the classification. Once a facial expression establishes classified deconstruction data, similar expressions within a certain threshold range will be automatically classified and deconstructed by the computer.

[0072] The audio deconstruction module automatically extracts and recognizes the dynamic voice information of personnel in real time based on the audio information collected by audio perception in the microspace, based on the training library support provided by the intelligent learning and training module, and selects, matches, and classifies and deconstructs the mixed and dynamic unstructured audio data into discrete and standardized structured data according to the voice classification in the training library and the pre-collected vocal membrane information of personnel, and pairs this data with the supervised personnel to support analysis, early warning and supervision work;

[0073] The panoramic three-dimensional management module mainly uses three-dimensional GIS technology to perform real-life three-dimensional modeling and management of the micro-space of the supervision site, and provides relevant data and functional interfaces to support other modules to call the three-micro-space scene data and functions, so as to realize the integrated application of various business functions in the three-dimensional scene;

[0074] The comprehensive portrait module is based on the basic information of the supervised persons, integrates the information processed by the image deconstruction module and the audio deconstruction module on a person-by-person basis, and supports the conduct of personnel behavior analysis and emotion analysis based on the intelligent learning and training module to achieve a panoramic real-time portrait of the individual personnel, and at the same time, a group portrait is conducted in a micro-space with the cell as the unit, reflecting the all-round status of individuals, groups, and micro-space scenes through comprehensive portraits;

[0075] The micro-space early warning and supervision module relies on the support of the intelligent learning and training module and the comprehensive portrait module. Based on real-time dynamic personnel portraits and environmental monitoring information, it conducts real-time monitoring and analysis of the individual and group conditions in the micro-space, and classifies and grades early warnings according to certain indicator thresholds, providing intelligent and automated information support for the comprehensive supervision work of managers.

[0076] The mechanism of classification and grading warning is: divided into two types of warning mechanisms: individual and group. Individual is based on personnel, and group is based on a single micro-space range. For individuals, it is based on the classification of facial expressions and body movements in real-time dynamic personnel portraits, combined with micro-space environmental factors to obtain the individual emotional comprehensive index value, and the index value is divided into four warning levels: green, orange, yellow, and red. Green represents stability without warning, orange represents instability, warning reminders, yellow represents easy excitement, warning attention at any time and voice warning, red represents excitement, direct alarm and automatic dispatch of manual disposal tasks. For groups, it is based on individuals, and the group emotional comprehensive index value is calculated based on the concentration state of grid units divided in the micro-space and the emotional comprehensive index of individuals in the concentrated area. The comprehensive calculation method can be the average calculation method, or the weighted average method with normal distribution. After the index value is calculated, the warning disposal is also carried out according to the above-mentioned four warning levels of green, orange, yellow, and red.

[0077] In this example, the ubiquitous perception module is composed of an image perception submodule, an audio perception submodule, an environment perception submodule, and a data aggregation and management submodule. Figure 2 As shown, where:

[0078] The image perception submodule includes functions such as camera and video data transmission and storage. In a micro space scene, generally two or more high-definition, wide-angle cameras are deployed at reasonable locations to achieve blind-angle video monitoring of the entire scene. These real-time collected image and video information are uniformly stored and managed through the video data transmission and storage function.

[0079] The audio perception submodule includes an audio perception device and audio data transmission, storage and other functions. In a micro-space scene, two or more high-fidelity, deep noise reduction sound collection devices are generally arranged at reasonable locations to achieve highly restored collection of voice information of people in the scene. These real-time collected audio information are uniformly stored and managed through audio data transmission and storage functions;

[0080] The environmental perception submodule includes a temperature perception device and environmental perception data transmission, storage and other functions. In a micro-space scene, a group of temperature and humidity perception devices are centrally deployed to realize the collection of environmental perception information. On this device, according to actual application needs, the perception units for monitoring other indicators can be continuously expanded. These real-time collected environmental perception information are uniformly stored and managed through data transmission and storage functions;

[0081] The data collection and management submodule provides management functions such as collection, integration, conversion channels, storage, and processing of various types of data. In addition to connecting to the image perception submodule, audio perception submodule, and environmental perception submodule, it also connects to external related business management systems to collect personnel and business-related information from external sources.

[0082] In this example, the intelligent learning and training module is composed of a feature dictionary classification submodule, an artificially extracted feature classification submodule, a feature deep learning classification submodule, a comprehensive analysis submodule, and a feature library management submodule, as shown in FIG3 , where:

[0083] The feature dictionary classification submodule divides and classifies image information and audio information based on pre-established dictionary libraries such as expressions, behaviors, voices, and objects, and realizes classification and extraction of image and audio features of human expressions, behaviors, voices, and objects in microspaces through multi-granular combined calculations;

[0084] The specific method of the dictionary classification method is as follows: for example, the pre-establishment of the facial image expression dictionary library of a person is to select a universal facial head image and divide the face into forehead, eyebrows, cheeks, cheekbones, nose, mouth and other parts, and then solidify various emotional facial expression parameters according to the change rate of the distance ratio between the facial parts under different emotions such as happiness, joy, tension, anger, etc. Each solidified expression part parameter forms a facial expression image dictionary item. In actual use, the facial image to be judged is compared with the dictionary item to judge the facial expression emotion classification of the person.

[0085] The manual feature extraction classification submodule includes functions such as sample selection, manual feature extraction, and model calculation. The image and audio information of certain people and objects are selected as analysis samples through samples, and then the features such as expression, behavior, voice, and objects in the sample information are extracted through the manual feature extraction function. Then, the calculation model constructed by algorithms such as regression and vector machines is used to automatically classify and extract implicit features of other images and audio information.

[0086] The process of manual feature extraction classification method is as follows: for example, the extraction of facial image expression features of personnel is to extract a certain proportion of sample personnel (such as 10% of the personnel sample) from the total number of detained personnel, and divide the facial expressions of these sample personnel into categories such as happiness, tension, joy, anger, etc. by manual judgment. On this basis, these facial images after the emotional types are divided into forehead, eyebrows, eyes, cheeks, cheekbones, nose, mouth and other parts, and the proportional changes in the distances between these parts of different samples under different emotional classifications are calculated, and then a set of interval range values ​​are obtained according to the emotional classification to form a sample reference value. In actual use, the facial expression image of the person to be judged is compared with the sample reference value to judge the facial expression emotional classification of the person.

[0087] The feature deep learning classification submodule is based on a deep network model and uses a machine learning algorithm to collect and summarize big data to establish a massive feature database of expressions, behaviors, voices, and objects. It then uses artificial intelligence to automatically classify and extract implicit features of image and audio information.

[0088] The process of the feature deep learning classification method is as follows: For example, the extraction of facial image expression features of personnel is to establish a massive feature database of facial emotion expression for the facial expression image data of the panoramic detainees and the non-detainees that can be collected. When judging each facial expression, the judgment result is obtained based on the mature machine algorithm and the big data matching calculation, and the judgment result is included in the feature big database. With the continuous accumulation of data in the big database, the accuracy of the big data matching calculation can be continuously improved. The feature extraction steps of the feature deep learning classification submodule are shown in Figure 4: First, the facial expressions, behaviors, and voices of the panoramic detainees are collected, and the image data of illegal items that may exist in the compulsory measures site are collected to establish a basic image and voice database, and then the mature facial, body behavior, voice, object and other image and voice machine learning recognition algorithms are used to extract and establish a large database of image and voice features corresponding to various emotions. When there are new detainees that need to be identified and classified, the classification and recognition results are obtained based on the established image and voice feature big database and the image and voice feature matching recognition method. These results are also stored in the image and voice feature big database to support subsequent analysis and judgment.

[0089] The comprehensive analysis submodule analyzes and calculates the classification features of the same object extracted by the feature dictionary classification submodule, the manually extracted feature classification submodule, and the feature deep learning classification submodule based on a multi-factor weighted calculation method, and obtains a comprehensive analysis result on the basis of retaining the three types of extraction results, and outputs it to the feature library management submodule. For the multi-factor weighted calculation method, dynamic weight allocation calculation is mainly performed based on factors such as sample size, timeliness, and degree of manual intervention, and continuous manual intervention can be performed based on actual analysis results; the comprehensive analysis submodule performs credibility analysis on the classification features of the same object extracted by the feature dictionary classification submodule, the manually extracted feature classification submodule, and the feature deep learning classification submodule based on factors such as sample size, timeliness, and intensity of manual intervention, and selects the result with the highest credibility on the basis of retaining the three types of extraction results and outputs it to the feature library management submodule. The steps of credibility analysis are: if the extraction results of the three sub-modules are the same, then this result is adopted; if the results of two of the sub-modules are the same and the other sub-module is different, then the former result is adopted; if the results of the three sub-modules are all different, then the result of the feature deep learning classification sub-module is selected, and at the same time this result is marked as a questionable result, and it is manually judged and corrected afterwards, and the corrected result is used as the accurate result, which is fed back to the feature dictionary classification sub-module as a dictionary item, fed back to the manual extraction feature classification sub-module as sample data, and fed back to the feature deep learning classification sub-module to enrich the image and speech feature big data.

[0090] The feature library management submodule implements the management function of the classification features of expressions, behaviors, voices, objects, etc. obtained through various classification extraction methods, including the input, output, storage, and retrieval functions of feature information.

[0091] In this example, the image deconstruction module is composed of a person expression analysis submodule, a person behavior analysis submodule, and an object analysis submodule. Figure 5 As shown, where:

[0092] The facial expression analysis submodule dynamically analyzes the facial expressions of people in the video perception image based on the support of the facial expression feature classification and extraction feature library of the intelligent learning and training module, and divides the facial expression features into two levels: one is to classify the facial expressions into positive and negative emotions according to the emotional extremes, and the other is to classify the facial expressions into happy, joyful, nervous, fearful, resentful, indifferent, angry, sad, etc. according to the multi-category emotions, and outputs the analysis results to the comprehensive portrait module;

[0093] The personnel behavior analysis submodule is based on the support of the behavior dynamic feature classification extraction feature library of the intelligent learning and training module, analyzes the personnel behavior dynamics in the video perception image, and also divides the behavior features through two levels. One is to classify it into two types of behavior classification, positive emotion and negative emotion, according to the emotion polarity classification, and the other is to classify it into happy, joyful, nervous, fearful, resentful, indifferent, angry, sad and other classified behaviors according to the multi-category emotion, and outputs the analysis results to the comprehensive portrait module;

[0094] The object analysis submodule is based on the support of the object feature classification and extraction feature library of the intelligent learning and training module, analyzes and tracks objects in the video perception image, focuses on accurate extraction and dynamic marking and tracking of prohibited items, and outputs the analysis results to the micro-space early warning and supervision module.

[0095] In this example, the audio deconstruction module is composed of an audio selection submodule, a vocal membrane matching submodule, and a speech analysis submodule. Figure 6 As shown, where:

[0096] The audio selection submodule selects the voice and other audio information emitted by the personnel from the mixed audio information based on the audio information in the microspace collected by the audio perception submodule and supported by the audio feature library of the intelligent learning and training module; the vocal membrane matching submodule accesses the vocal membrane information of the personnel in the data aggregation and management submodule, and based on the mature vocal membrane matching algorithm, pairs and analyzes the voice information of the personnel selected in the audio selection submodule with the vocal membrane information of the personnel, and binds the perceived voice to the personnel;

[0097] The speech analysis submodule is based on the audio feature classification and feature library support of the intelligent learning and training module, analyzes the personnel speech information in the audio perception data, and divides the personnel speech features through two levels, one is to classify the speech into positive emotion and negative emotion according to the emotional polarity, and the other is to subdivide the speech into happy, joyful, nervous, fearful, resentful, indifferent, angry, sad and other classified speech according to multiple categories of emotions, and outputs the analysis results to the comprehensive portrait module;

[0098] In this example, the panoramic 3D management module is composed of a GIS engine submodule, a micro space model management submodule, a micro space grid management submodule, a micro space scene management submodule, and a panoramic 3D service interface submodule. Figure 7 As shown, the GIS engine submodule adopts mature 3D GIS commercial software, and provides basic GIS functions such as spatial data management, analysis, and calculation through secondary development and integration; the micro-space model management submodule is based on the basic functions of the GIS engine submodule to realize the functions of adding, deleting, modifying, and encoding 3D models in the micro-space;

[0099] The micro-space grid management submodule is based on the basic functions of the GIS engine submodule, and divides the micro-space plane into reasonable grids, and manages the grid units formed by the division. The division of the micro-space grid is mainly convenient for the spatial correlation analysis of personnel behavior activities and emotions; the micro-space scene management submodule is based on the basic functions of the GIS engine submodule, and manages the addition, deletion, encoding, etc. of various GIS space objects such as icons, texts, graphics, and annotations in the micro-space except for the three-dimensional model objects;

[0100] The three-dimensional service interface submodule uses common format interfaces such as Web services and Rest services to open up GIS basic functions, micro-space model data, micro-space grid data, micro-space scene data and other functions or data in an interface componentized manner to support the integrated call of other functional modules of the system;

[0101] In this example, the comprehensive portrait module is composed of a personnel basic information management submodule, a behavior and emotion analysis submodule, a personnel portrait submodule, and a micro-space portrait submodule. Figure 8 As shown, where:

[0102] The basic personnel information management submodule manages the basic information of the supervised personnel in a unified manner by aggregating the personnel information of the external business system, including functions such as adding, modifying, deleting, and synchronizing the basic personnel information; the behavior and emotion analysis submodule integrates the classification information of the personnel expressions, behaviors, and voice features analyzed by the received image deconstruction module and the audio deconstruction module, and performs a comprehensive analysis in combination with the micro-space environment status and the basic personnel information to obtain the behavior and emotion analysis results of the supervised personnel as individuals;

[0103] The personnel portrait submodule performs portrait management on the supervised personnel based on the analysis results of the behavior and emotion analysis submodule, including static portrait and dynamic portrait. The static portrait is a characterization of the personality, emotion, and behavior characteristics of the personnel based on the basic information of the personnel and the results of long-term dynamic monitoring and analysis of the personnel, while the dynamic portrait is a characterization of the real-time emotion and behavior characteristics of the personnel based on the real-time dynamic information and analysis results of ubiquitous perception. The micro-space portrait submodule takes a single micro-space as a unit and the supervised personnel group in each micro-space as a whole to portrait the group behavior, emotion, and stability changes.

[0104] In this example, the micro-space warning and supervision module is composed of a personnel abnormal warning submodule, a group abnormal warning submodule, a prohibited item dynamic monitoring submodule, and a comprehensive supervision submodule. Fig. 9 As shown, where:

[0105] The personnel abnormal warning submodule divides the personnel abnormal conditions into four levels: green, orange, yellow and red, representing four state intervals: stable, relatively unstable, easily excited and excited, respectively, according to the personnel dynamic portrait information of the comprehensive portrait module through threshold adjustment. The management personnel makes supervision decisions based on the abnormal warning information;

[0106] The group abnormality warning submodule uses a single cell micro-space as a unit, monitors and analyzes the group abnormality based on the comprehensive portrait module and the dynamic information of the personnel abnormality warning submodule, and divides the group abnormality into four levels: green, orange, yellow, and red through threshold adjustment, representing the four state intervals of group stability, group instability, group prone to radicalism, and group radicalism, respectively. The management personnel make supervision decisions based on the abnormal warning information;

[0107] The prohibited items dynamic detection submodule dynamically detects and tracks prohibited items that may exist in the micro space based on the support of the analysis function of the intelligent learning and training module;

[0108] The comprehensive supervision submodule uses the three-dimensional GIS scene as a carrier to carry out comprehensive and visual supervision of people and groups in the microspace, including indicator monitoring and analysis, microspace status inspection, personnel information query, cell scheduling, personnel allocation management, sensing equipment status monitoring and other functions, and connects with external business systems to output relevant analysis results to external business systems to support the operation and management of other business systems.

[0109] The operation process of each functional module in this system is as follows: the ubiquitous perception module is the input source of the entire system, responsible for collecting various types of perception information and aggregating external business system information, and outputting the perception data to the intelligent learning and training module, image deconstruction module, and audio deconstruction module. The intelligent learning and training module supports the analysis and calculation of the image deconstruction module and the audio deconstruction module, and the analysis and calculation results of the image deconstruction module and the audio deconstruction module will be fed back to the intelligent learning and training module to supplement the feature library information, and the analysis and calculation accuracy of the intelligent learning and training module will be continuously improved through machine learning algorithms. The analysis results of the image deconstruction module and the audio deconstruction module are combined with the environmental perception information of the ubiquitous perception module and converged to the comprehensive portrait module to realize the comprehensive portrait management of personnel and microspaces. The comprehensive portrait module The portrait results are output to the micro-space early warning and supervision module to support comprehensive and visualized early warning analysis and supervision. The micro-space early warning and supervision module is the display, use and output terminal of the entire system. Management police officers control the module based on PC, mobile APP and command and dispatch large screen to realize the dispatch and use of the entire system, and connect with external systems to output management, analysis and decision-making information to external business systems. The panoramic three-dimensional management module provides basic three-dimensional GIS capabilities to support the operation of intelligent learning and training modules, comprehensive portrait modules and micro-space early warning and supervision modules, so as to effectively solve the real-time and all-round supervision problems of personnel and personnel behaviors in micro-space environments of various compulsory measures places, improve supervision efficiency, reduce management costs and improve service levels.

[0110] The technical solution provided by the present invention is described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A panoramic portrait intelligent supervision system for personnel behavior in micro-spaces, characterized by: It includes a ubiquitous perception module, an intelligent learning and training module, an image deconstruction module, an audio deconstruction module, a panoramic three-dimensional management module, a comprehensive portrait module, and a micro-space early warning and supervision module, wherein: the ubiquitous perception module is used to comprehensively perceive and collect information on the environmental status of the micro-space of the supervision site, and access the external information of the business management system related to the supervised personnel, so as to realize the real-time, comprehensive perception, collection and storage management of the supervised personnel's behavior activities and spatial environment conditions; the intelligent learning and training module is used to classify, parse and identify the audio and video information collected by the ubiquitous perception module, analyze and summarize the emotional characteristics of the personnel through the trained personnel images, voices, and behavioral features, and manage and verify the learning and training results; the image deconstruction module includes a personnel expression analysis submodule, a personnel behavior analysis submodule, and an item analysis submodule, wherein: the personnel expression analysis submodule is used to use the facial expression feature classification and extraction feature library provided by the intelligent learning and training module to classify the ubiquitous The facial expressions of people in the video perception images collected by the perception module are dynamically analyzed, and the analysis results are output to the comprehensive portrait module; the personnel behavior analysis submodule is used to adopt the behavior dynamic feature classification provided by the intelligent learning and training module to extract the feature library, analyze the personnel behavior dynamics in the video perception images collected by the ubiquitous perception module, and output the analysis results to the comprehensive portrait module; the object analysis submodule is used to adopt the object feature classification provided by the intelligent learning and training module to extract the feature library, analyze and track the objects in the video perception images collected by the ubiquitous perception module, focus on accurately extracting and dynamically marking and tracking prohibited objects, and output the analysis results to the micro-space early warning and supervision module; the personnel expression analysis submodule also divides the expression features through two levels, one is to classify them into two categories of positive emotions and negative emotions according to the emotional extreme classification, and the other is to classify expressions into happy, joyful, nervous, fearful, resentful, indifferent, angry, and sad according to multiple categories of emotions; The personnel behavior analysis submodule also divides the behavior characteristics through two levels. One is to classify the behavior into two categories of positive emotion and negative emotion according to the emotion polarity classification, and the other is to classify the behavior into happy, joy, tension, fear, resentment, indifference, anger, and sadness according to the multi-category emotion classification. The image deconstruction module is used for the video image information collected by the ubiquitous perception module, based on the training library support provided by the intelligent learning and training module, to automatically extract and identify the facial features, behavior dynamics and various items in the supervision place in real time, and classify and deconstruct the extracted and identified elements according to the image classification in the training library;The audio deconstruction module automatically extracts and recognizes the dynamic voice information of personnel in real time based on the audio information collected by the ubiquitous perception module and the training library support provided by the above-mentioned intelligent learning and training module, and selects, matches, classifies and deconstructs the voice information according to the voice classification in the training library and the pre-collected vocal membrane information of personnel, and pairs the obtained data with the supervised personnel; the panoramic three-dimensional management module is used to perform real-life three-dimensional modeling and management of the micro-space of the supervision site based on the three-dimensional GIS technology; the comprehensive portrait module includes a personnel basic information management submodule, a behavior and emotion analysis submodule, a personnel portrait submodule, and a micro-space portrait submodule, wherein: the personnel basic information management submodule This information management submodule is used to uniformly manage the basic information of supervised personnel by aggregating personnel information from external business management systems; the behavior and emotion analysis submodule is used to integrate the personnel expression, behavior, and voice feature classification information analyzed by the received image deconstruction module and audio deconstruction module, and conduct a comprehensive analysis based on the micro-space environment status and basic personnel information to obtain the behavior and emotion analysis results based on the supervised personnel as individuals; the personnel portrait submodule is used to use the behavior and emotion analysis results to manage the supervised personnel as portraits; the micro-space portrait submodule is used to use a single micro-space as a unit to manage the supervised personnel group in each micro-space as a whole. As a whole, the group behavior, emotions and stability changes are portrayed. The personnel portrait submodule manages the portraits of the supervised personnel in two ways: static portrait and dynamic portrait. The static portrait is a portrayal of the personality, emotions and behavior characteristics of the personnel based on the basic information of the personnel and the results of long-term dynamic monitoring and analysis of the personnel. The dynamic portrait portrays the real-time emotions and behavior characteristics of the personnel based on the real-time dynamic information and analysis results of ubiquitous perception. The comprehensive portrait module is used to integrate the information processed by the image deconstruction module and the audio deconstruction module based on the basic information of the supervised personnel and the personnel as a unit, and based on the intelligent learning and training model The training library provided by the module supports the conduction of personnel behavior analysis and emotion analysis to realize the panoramic real-time portrait of individual personnel, and at the same time, the group portrait is carried out in the micro-space scene with the cell as the unit, and the comprehensive portrait reflects the all-round status of individuals, groups, and micro-space scenes; the micro-space early warning and supervision module includes a personnel abnormal early warning submodule, a group abnormal early warning submodule, a contraband dynamic monitoring submodule, and a comprehensive supervision submodule, wherein: the personnel abnormal early warning submodule is used to divide the personnel abnormal status into four levels of green, orange, yellow, and red through threshold adjustment according to the personnel dynamic portrait information of the comprehensive portrait module, representing the four state intervals of stable, relatively unstable, easily excited, and excited respectively;The group abnormal warning submodule is used to monitor and analyze the abnormal situation of the group based on the dynamic information of the comprehensive portrait module and the personnel abnormal warning submodule, taking a single cell microspace as a unit, and divide the abnormal situation of the group into four levels of green, orange, yellow and red through threshold adjustment, which respectively represent the four state intervals of collective stability, collective instability, collective extremism and collective extremism; the prohibited items dynamic detection submodule is used to dynamically detect and track the prohibited items that may exist in the microspace; the comprehensive supervision submodule uses the three-dimensional GIS scene as a carrier to carry out comprehensive and visual supervision of personnel and groups in the microspace, and connects with the external business management system to output the relevant analysis results to the external business system; the microspace warning and supervision module is used to rely on the data of the intelligent learning and training module and the comprehensive portrait module as support, based on the real-time dynamic personnel portrait and environmental monitoring information, to monitor and analyze the individual and group situations in the microspace in real time, and to carry out classified and graded warnings. ; 2. The panorama portrait intelligent supervision system for personnel behavior in micro-space according to claim 1 is characterized by: The ubiquitous perception module includes an image perception submodule, an audio perception submodule, an environmental perception submodule, and a data aggregation and management submodule, wherein: the image perception submodule is used for monitoring the entire scene of the micro-space of the supervision venue without blind spots, and storing and managing the obtained video image data; the audio perception submodule is used for highly restored collection of voice information of personnel in the micro-space of the supervision venue, and storing and managing the collected audio information; the environmental perception submodule is used for collecting temperature and humidity data in the micro-space of the supervision venue, and storing and managing the collected environmental perception information; the data aggregation and management submodule is used to collect, integrate, convert channels and store and manage various types of data provided by the image perception submodule, audio perception submodule and environmental perception submodule, and connect with external related business management systems to collect personnel and business information from external sources.

3. The micro-space-oriented personnel behavior panoramic portrait intelligent supervision system according to claim 1 is characterized by: The intelligent learning and training module includes a feature dictionary classification submodule, an artificially extracted feature classification submodule, a feature deep learning classification submodule, a comprehensive analysis submodule, and a feature library management submodule, wherein: The feature dictionary classification submodule is used to use a pre-established dictionary library to divide and classify image information and audio information, and to achieve classification and extraction of image and audio features of human expressions, behaviors, voices, and objects in microspaces through multi-granularity combined calculations; The manually extracted feature classification submodule is used to select certain personnel images, object images, and audio information as analysis samples, manually extract facial expressions, behaviors, voices, and object features in the analysis samples, and use an algorithm model to automatically classify and extract implicit features of other images and audio information except the analysis samples; the feature deep learning classification submodule is used to establish a massive feature database of facial expressions, behaviors, voices, and objects through big data collection and induction, and then automatically classify and extract implicit features of image and audio information; the comprehensive analysis submodule is used to use a multi-factor weighted calculation method to analyze and calculate the classification features of the same object extracted by the feature dictionary classification submodule, the manually extracted feature classification submodule, and the feature deep learning classification submodule, and obtain a comprehensive analysis result on the basis of retaining the three types of extraction results, and then output it to the feature library management submodule; the feature library management submodule is used to realize the management function of the classification feature data output by the comprehensive analysis submodule.

4. The micro-space-oriented personnel behavior panoramic portrait intelligent supervision system according to claim 1 is characterized by: The audio deconstruction module includes an audio selection submodule, a vocal membrane matching submodule, and a speech analysis submodule, wherein: the audio selection submodule is used to use the audio feature library provided by the intelligent learning and training module to select the voice and other audio information emitted by the person from the audio information collected by the ubiquitous perception module; the vocal membrane matching submodule is used to access the vocal membrane information of the person extracted by the ubiquitous perception module, pair and analyze the person's voice information selected in the audio selection submodule with the person's vocal membrane information, and bind the perceived voice to the person; the speech analysis submodule is used to extract the feature library according to the audio feature classification provided by the intelligent learning and training module, analyze the person's voice information in the audio information, and output the analysis results to the comprehensive portrait module after classifying the person's voice features.

5. The panorama portrait intelligent monitoring system for personnel behavior in micro-space according to claim 1 is characterized by: The panoramic three-dimensional management module includes a GIS engine submodule, a microspace model management submodule, a microspace grid management submodule, a microspace scene management submodule, and a panoramic three-dimensional service interface submodule, wherein: the GIS engine submodule is used to provide management, analysis, and calculation of spatial data; the microspace model management submodule realizes the addition, deletion, modification, and encoding of three-dimensional models in the microspace based on the basic functions of the GIS engine submodule; the microspace grid management submodule grid-segments the microspace plane based on the basic functions of the GIS engine submodule, and manages the grid units formed by the segmentation; the microspace scene management submodule adds, deletes, and manages the GIS space objects other than the three-dimensional model objects in the microspace based on the basic functions of the GIS engine submodule; and the three-dimensional service interface submodule is used to open the basic functions of the GIS engine submodule, microspace model data, microspace grid data, and microspace scene data in an interface componentized manner in a universal format interface.

Images