GIS artificial intelligence CCTV map-based data visualization city operation management system and server

Abstract

This GIS artificial intelligence CCTV map-based data visualization city operation management system comprises: a CCTV for generating a monitoring image for each area of a city; an IOT sensor for sensing city data that is data on city elements including traffic, environment, population, and energy; a server for collecting the city data from the IOT sensor, detecting a dangerous situation via recognition of an object included in the image collected from the CCTV, extracting associated data on the detected dangerous situation, and visualizing a CCTV image analysis result; and an administrator terminal for outputting the CCTV image analysis result of the server and a visualization result of the city data.

Description

GIS AI CCTV map-based data visualization city operation management system and server

[0001] The present disclosure relates to a GIS artificial intelligence CCTV map-based smart city visualization monitoring system and server, and more specifically, to a system and server that provides a map playback service that confirms and simulates the flow of data by linking GIS map information and a specific situation.

[0002]

[0003] Unless otherwise indicated herein, the materials described in this section are not prior art to the claims of this application, and their inclusion in this section is not intended to be admitted as prior art.

[0004] An ICT (Information and Communications Technology) smart city is a concept that integrates information and communication technology (ICT) into a city. It is a model that integrates big data technology to solve various urban problems and improve the quality of life. Smart cities provide infrastructure data services and are emerging as a key element in addressing urban issues caused by rapid urbanization and the serious challenges of climate change. While many systems are currently utilized for urban management, the most common CCTV-based system is the Client-Server (CS) solution, which can display multiple images in real time. A Video Management System (VMS) offers the advantage of allowing the real-time display of multiple high-quality images. However, VMS requires constant human monitoring, requiring manual confirmation of event occurrences. Furthermore, these city control systems are developed as standalone solutions that maximize transmission efficiency by directly communicating with the server to circumvent limitations of video transmission protocols, making integration with and expansion of external systems difficult. Furthermore, existing city control systems struggle to identify abnormal situations occurring at specific times and to track data flow.

[0005] In particular, a major drawback of existing video monitoring systems is that they require constant monitoring of the footage. Consequently, when a crime or missing child is discovered, personnel must manually review footage and review previously recorded footage, consuming significant manpower and time. This often leads to missed opportunities to address urban issues that require rapid resolution.

[0006] [Prior Art Literature]

[0007] [Patent Document]

[0008] (Patent Document 1) 1. Korean Patent Registration No. 10-0993729 (November 4, 2010)

[0009] (Patent Document 2) 2. Korean Patent Registration No. 10-1336317 (November 26, 2013)

[0010]

[0011] The GIS AI CCTV map-based data visualization city operation management system and server according to the embodiment applies algorithms and thresholds to real-time images of all CCTVs located on the GIS CCTV AI map and integrates data occurring in the city to confirm data centered on the location where an event occurred and enable identification of the cause of the event.

[0012] In addition, the CCTV artificial intelligence model according to the embodiment repeatedly applies deep learning-based learning to accumulated city video data in the past to directly recognize the face of a real-time monitoring target or recognize and track an object from real-time CCTV video.

[0013] In addition, through examples, we provide a data-centric control service that utilizes the intuitiveness of GIS to quickly identify the cause, track the situation, and take early action on countermeasures based on the location provided by web-based maps.

[0014] The GIS artificial intelligence CCTV map-based data visualization urban operation management system and server according to the embodiment extracts video data and urban data of the location where a problem occurred by specifying a specific point in time, and replays the situation before and after the problem occurred by fusing the metadata of the extracted data, thereby facilitating the tracking of data flow and enabling the manager to respond to the problem situation quickly and accurately.

[0015] Additionally, by gathering data collected from various points and providing visualizations, it maximizes insight into the situation and supports decision-making.

[0016] However, the problems to be solved according to one embodiment are not limited to those mentioned above.

[0017]

[0018] The GIS artificial intelligence CCTV map-based data visualization urban operation management system according to the embodiment includes CCTV that generates monitoring images for each district of the city; IOT sensors that sense urban data, which is data on urban elements including traffic, environment, population, and energy;

[0019] A server that collects city data from IoT sensors, detects dangerous situations through object recognition included in images collected from the CCTV, extracts related data regarding the detected dangerous situations, and visualizes the CCTV image analysis results; and an administrator terminal that outputs the server's CCTV image analysis results and the visualization results of the city data.

[0020]

[0021] The GIS AI CCTV map-based data visualization urban operations management system and server described above provides a data-driven control service that leverages the intuitive nature of GIS to quickly identify the cause, track issues, and initiate early response measures. This allows city managers to immediately recognize problems that arise, accurately identify the cause based on location information, and take swift action.

[0022] In addition, the GIS artificial intelligence CCTV map-based data visualization urban operation management system and server according to the embodiment improves the efficiency of data-based urban management by providing objective data on the area where a problem occurred or the problem that occurred along with the video.

[0023] The effects of the present invention are not limited to the effects described above, and should be understood to include all effects that can be inferred from the detailed description of the present invention or the composition of the invention described in the claims.

[0024]

[0025] Figure 1 is a diagram showing a GIS artificial intelligence CCTV map-based data visualization urban operation management system according to an embodiment.

[0026] Figure 2 is a drawing showing the data processing configuration of a server (100) according to an embodiment.

[0027] Figure 3 is a drawing showing an example of an output of an event visualization unit according to an embodiment.

[0028] Figure 4 is a diagram showing the visualization results of location-based data linked to a GIS map according to an embodiment.

[0029]

[0030] Hereinafter, the embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Regardless of the drawing numbers, identical or similar components will be given the same reference numbers, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably only for the convenience of writing the specification, and do not in themselves have distinct meanings or roles. In addition, when describing the embodiments disclosed in this specification, if it is determined that a specific description of a related known technology may obscure the gist of the embodiments disclosed in this specification, a detailed description thereof will be omitted. In addition, the attached drawings are only intended to facilitate easy understanding of the embodiments disclosed in this specification, and the technical ideas disclosed in this specification are not limited by the attached drawings, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and technical scope of the present invention.

[0031] Terms that include ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another.

[0032] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0033] In this application, terms such as “include” or “have” are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but should be understood not to exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0034] In this specification, the term "unit" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Furthermore, a single unit may be realized using two or more pieces of hardware, and two or more units may be realized by a single piece of hardware.

[0035] Some of the operations or functions described herein as being performed by a terminal, apparatus, or device may instead be performed by a server connected to the terminal, apparatus, or device. Similarly, some of the operations or functions described herein as being performed by a server may also be performed by a terminal, apparatus, or device connected to the server.

[0036] Hereinafter, the present invention will be described in detail with reference to the attached drawings.

[0037] Figure 1 is a diagram showing a GIS artificial intelligence CCTV map-based data visualization city operation management system according to an embodiment.

[0038] Referring to FIG. 1, a GIS artificial intelligence CCTV map-based data visualization urban operation management system according to an embodiment may be configured to include a server (100), a CCTV (200), an IoT sensor (300), and an administrator terminal (400). The CCTV (200) is installed in alleys, roads, and major facilities of a city district to generate monitoring images for each area. The IoT sensor (300) collects urban data. In an embodiment, the urban data may include data on urban elements including traffic, environment, population, and energy as a series of data generated in the city. The server (100) collects urban data from the IoT sensor (300) and detects a dangerous situation through object recognition included in the image collected from the CCTV (200). Thereafter, the server (100) extracts related data for the detected dangerous situation and visualizes the CCTV image analysis results. The administrator terminal (400) outputs the CCTV image analysis results of the server and the visualization results of the urban data.

[0039] The GIS AI CCTV map-based data visualization urban operation management system, according to the embodiment, pre-trains on previously recorded video data and applies algorithms to real-time video based on the learning results. This allows it to recognize objects' faces and immediately notify managers if identical information is detected in other areas when the object moves. This allows the system that managers monitor to implement data-driven control.

[0040] In addition, the GIS artificial intelligence CCTV map-based data visualization urban operation management system according to the embodiment analyzes urban data including data from each site together with real-time video to identify problem situations and quickly identify the cause of the problem situation.

[0041] Additionally, the GIS AI CCTV map-based data visualization urban operation management system, according to the embodiment, links monitoring footage and city data with relevant organizations related to disasters such as fires or crimes and terrorism. In the embodiment, relevant organizations include, but are not limited to, fire departments, police stations, hospitals, and government offices.

[0042] Furthermore, the GIS AI CCTV map-based data visualization urban operations management system, according to the embodiment, can be integrated with the city's Intelligent Transport System (ITS) to serve as a transportation network management system. Furthermore, it can be expanded into a CPS (Participation System) that encourages citizen participation tailored to each location or situation managed by the city, thereby implementing a comprehensive smart city system that communicates and manages the entire city.

[0043] FIG. 2 is a diagram showing a data processing configuration of a server (100) according to an embodiment.

[0044] Referring to FIG. 2, a server (100) according to an embodiment may be configured to include a data collection unit (110), a metadata generation unit (120), a metadata learning unit (130), a problem situation event detection unit (140), an event visualization unit (150), and a feedback unit (160). The term 'unit' used in this specification should be interpreted to include software, hardware, or a combination thereof, depending on the context in which the term is used. For example, the software may be machine language, firmware, embedded code, and application software. As another example, the hardware may be a circuit, a processor, a computer, an integrated circuit, an integrated circuit core, a sensor, a MEMS (Micro-Electro-Mechanical System), a passive device, or a combination thereof.

[0045] The data collection unit (110) collects real-time video data and training data from CCTV. The real-time video data may include monitoring video collected in real time from CCTV. The training data is training data for an artificial neural network model used for problem situation event detection and data analysis. In an exemplary embodiment, the training data may include, but is not limited to, previous monitoring video, data characteristics of the problem situation, risk levels by data characteristics, and problem situation scenarios.

[0046] The metadata generation unit (120) auto-labels the collected image data to build big data and trains an object recognition model using the big data. To this end, the metadata generation unit (120) receives training data and real-time image data from the collection unit (110). In an embodiment, the data transmitted includes various characteristics and situations of objects to be used for training. Thereafter, the metadata generation unit (120) auto-labels the collected image data. Auto-labeling is the task of identifying objects in images or videos and assigning labels to the identification results. The metadata generation unit (120) can perform auto-labeling using computer vision technology. Thereafter, the metadata generation unit (120) builds a big data set using the auto-labeled data. Thereafter, the metadata generation unit (120) trains an object recognition model using the built big data. The metadata generation unit (120) trains and optimizes a neural network model using deep learning technology. The trained model is fine-tuned to detect and identify objects in real-time images.

[0047] Additionally, the metadata generation unit (120) preprocesses unstructured data, which is urban data, and constructs metadata by fusing the preprocessed unstructured data with the image data processing results. In the embodiment, the unstructured urban data is data collected through IoT sensors and includes various forms of data, such as text, images, and audio.

[0048] The metadata generation unit (120) first performs preprocessing of the collected urban data to construct metadata. For example, the metadata generation unit (120) performs text preprocessing such as tokenization, stopword removal, and stem extraction for text data, and performs preprocessing appropriate for the corresponding data format for image or audio data.

[0049] In addition, in the embodiment, the metadata generation unit (120) collects CCTV video data and city data, and performs auto-labeling through learning on the collected data to convert unstructured data into metadata.

[0050] Additionally, in the embodiment, the metadata generation unit (120) processes image data applied to metadata by applying an artificial intelligence-based object recognition algorithm. In the embodiment, the image processing process may include processes such as object recognition, classification, and segmentation, which may be performed through an object recognition algorithm. In the embodiment, the object recognition algorithm may be based on at least one of a facial recognition algorithm, an object recognition algorithm, and a crowd behavior recognition algorithm.

[0051] In an embodiment, a face recognition algorithm recognizes people and faces by applying face recognition pipeline technology. In an embodiment, the face recognition algorithm detects facial objects in input data (Face Detection), generates bounding boxes, and performs non-maximum suppression to select the most reliable box among overlapping bounding boxes. Then, landmark locations are identified in the selected bounding boxes (landmark locations), and major landmarks or feature points of the face are accurately detected and adjusted (Face alignment). Afterwards, facial features are extracted (feature extraction) and classified, thereby recognizing the face. In an embodiment, the face recognition algorithm includes, but is not limited to, deep learning-based algorithms (e.g., Deep Face, Open Face, Arc Face), Local Binary Pattern Histogram (LBPH), and Multi-task Cascaded Convolutional Networks (MTCNN).

[0052] In one embodiment, the object recognition algorithm detects the number and actions of hazardous materials and human objects. Furthermore, the object recognition algorithm can recognize hazardous materials and specific objects in situations where multiple objects are mixed. In one embodiment, the object recognition algorithm, based on artificial intelligence learning and algorithmic technologies, can recognize specific objects among multiple objects and recognize each object as a distinct entity. Furthermore, object tracking is performed using an algorithm that recognizes objects such as hazardous materials.

[0053] Crowd behavior recognition algorithms segment objects and analyze the density and behavioral patterns of those densely packed in a specific location. For example, crowd behavior recognition algorithms perform crowd and person detection, crowd and person tracking, feature extraction, behavior classification, and anomaly detection. In one embodiment, a crowd behavior recognition algorithm recognizes each object in an image containing multiple people as a distinct person.

[0054] In the embodiment, the metadata generation unit (120) recognizes crowds and human objects through the algorithm described above, and tracks the recognized crowds and human objects. Thereafter, the metadata generation unit (120) periodically extracts features of the tracked crowds and human objects while tracking the crowds and human objects. In the embodiment, the extracted features may include, but are not limited to, density, collectiveness, speed, direction, arousal, valence, etc. In the embodiment, the metadata generation unit (120) may include, but are not limited to, congestion, fight, fail, contraflow pedestrian, panic, fire, accident, etc.

[0055] Thereafter, the metadata generation unit (120) fuses urban data and image data to provide a large amount of information about specific events, locations, or objects. The metadata generation unit (120) then uses the fused data to construct metadata. In one embodiment, the metadata stores all information about the city's situation, environment, events, etc. For example, the metadata may be generated by combining urban data about an event occurring in a specific area with image data from that area.

[0056] The metadata generated in this example is utilized in a variety of applications. For example, it enables metadata-based decision-making in areas such as urban planning, traffic analysis, and emergency response.

[0057] The metadata learning unit (130) sets a threshold according to urban management rules through the constructed metadata and creates a problem situation event detection model based on a rule engine.

[0058] For example, the metadata learning unit (130) analyzes the constructed metadata to identify patterns, characteristics, relationships, etc. of urban data and image data. In an embodiment, metadata analysis is a process for obtaining information about specific urban situations or events. Thereafter, the metadata learning unit (130) establishes urban management rules based on the analysis results of urban data and image data. In an embodiment, the urban management rules include conditions, thresholds, behavioral rules, etc. for specific situations or events. For example, rules may include rules for traffic congestion, environmental pollution, safety issues, etc. Thereafter, a threshold is defined for each situation or event according to the established urban management rules. In an embodiment, the threshold is a reference value that is considered a problematic situation when a specific condition is met. In an embodiment, the threshold may be adjusted according to the characteristics and risk level of the metadata. Thereafter, the metadata learning unit (130) generates a problem situation event detection model based on the established urban management rules and thresholds. In an embodiment, the problem situation event detection model is a rule engine that recognizes problem situations through analysis of urban data and monitoring images. In an embodiment, the metadata learning unit (130) may generate the problem situation event detection model based on the rule engine. A rule engine is a rule-based system, an algorithm that evaluates input data and detects problem situations based on defined rules.

[0059] The problem situation event detection unit (140) detects problem situations by inputting real-time metadata into the generated problem situation event detection model. In the embodiment, the problem situation event detection unit (140) monitors city data in real time through the problem situation event detection model and detects problem situations according to set rules and thresholds.

[0060] In the embodiment, the problem situation event detection unit (140) calculates a rule violation value from CCTV footage and city data input in real time, and detects a problem situation event when the calculated rule violation value exceeds the caution stage threshold.

[0061] In an embodiment, the problem situation event detection unit (140) calculates a rule violation value by applying a set rule to the collected data monitoring results. For example, the problem situation event detection unit (140) calculates a rule violation value by analyzing objects recognized from CCTV video data, CCTV video input in real time, and city data. For example, the problem situation event detection unit (140) can identify related information for the detected problem situation event and calculate a rule violation value based on the related information. Specifically, when the problem situation event detection unit (140) detects a traffic congestion event, it identifies related information such as vehicle density, moving speed, and whether or not a signal was violated from the traffic data and evaluates the related information. Thereafter, it can calculate a rule violation value based on the result of the related information evaluation. In addition, when the problem situation event detection unit (140) detects a fall event of a facility object, it can calculate a rule violation value proportional to each piece of related information, such as the size of the facility object, the density of the surrounding crowd, and the size of the area where the event occurred. In addition, when a problem situation event detection unit (140) detects a fight event, it can determine the crowd density, presence or absence of a weapon, bleeding and degree of injury of the person participating in the fight event as related information, and calculate a rule violation number based on the related information.

[0062] Thereafter, the problem situation event detection unit (140) verifies whether the calculated rule violation value exceeds the set caution level threshold. Thereafter, the problem situation event detection unit (140) verifies whether an event exceeding the caution level threshold is a problem situation event.

[0063] In addition, the problem situation event detection unit (140) transmits an alarm to a preset administrator terminal and a public institution terminal or server when the calculated rule violation number exceeds a preset emergency level threshold.

[0064] The event visualization unit (150) performs visualization of GIS map linkage according to detected problem situation events and performs map replay according to the problem situation. For example, if a fight event is detected, the event visualization unit (150) can close up a person object participating in the fight, recognize the identity, output the identity recognition result, and repeatedly output the fight event.

[0065] In addition, when a problem situation event occurs, the event visualization unit (150) visualizes location-based data linked to a GIS map by fusing CCTV footage and city data linked to the problem situation event.

[0066] Fig. 3 is a drawing showing an example of an output of an event visualization section according to an embodiment.

[0067] Referring to FIG. 3, the event visualization unit (150) according to the embodiment can output the location of the human object and the monitoring video of the human object when a human object with a rule violation count exceeding a certain level is detected. Furthermore, the facial recognition results and identification information of the object can be output together. Furthermore, the event visualization unit (150) according to the embodiment can output accident status and detailed statistical data by city.

[0068] FIG. 4 is a diagram showing the visualization result of location-based data linked to a GIS map according to an embodiment. Referring to FIG. 4, in an embodiment, when there is tracking object information to be detected, the event visualization unit (150) compares the tracking object information with a facial area extracted from a CCTV video to generate tracking information, and visualizes and outputs the generated tracking information. In an embodiment, the tracking information may include the movement line, location and movement information of the tracking object extracted from accumulated city data and video data, GPS coordinates of the location where the tracking object information was found, the time of discovery, the movement path and expected location of the tracking object, etc. In an embodiment, when the problematic situation is a crime, an object corresponding to the subject of the crime in a video in which the problematic situation is recorded may be identified as a tracking object, and the identified object may be tracked to generate and output tracking information.

[0069] In addition, in the embodiment, when a user terminal or administrator terminal designates a specific point in time when a problem situation occurs, the event visualization unit (150) performs playback that replays the situation before and after the problem situation occurs by fusing video data and city data of a location associated with the designated problem situation. To this end, the event visualization unit (150) inputs the designated problem situation and specific point in time into the problem situation event detection model, recognizes key objects included in the designated problem situation, and extracts video data and city data of all locations including the key objects. Thereafter, the extracted video data and city data can be arranged according to a timeline, and feature events and tracked objects can be output in close-up on the timeline.

[0070] In addition, in the embodiment, when a problem situation event is detected, the event visualization unit (150) inputs metadata related to the problem situation event into a risk classification model that classifies the risk of the problem situation event, and when the risk falls within a preset risk level, calls all adjacent CCTVs in the area where the problem situation event occurred, collects additional CCTV footage from the called CCTVs, and outputs the collected footage.

[0071] For example, when information on a tracking object to be detected is input, the event visualization unit (150) compares the tracking object information with a facial region extracted from CCTV footage to generate tracking information, and extracts the expected location of the tracking object at the next time using the tracking information. Thereafter, an expected route including the expected location is calculated, and a CCTV adjacent to the calculated expected route is called to set the tracking object to be tracked in advance, and additional information can be acquired and output from the CCTV adjacent to the expected route. In an embodiment, when a high-risk situation including kidnapping, weapon recognition, or criminal recognition occurs, a scene where the risk situation occurred is extracted, and a key object is specified in the extracted scene. Thereafter, a face corresponding to the key object is set as a tracking object, and tracking information for the set tracking object can be generated. In an embodiment, the additional CCTV footage includes tracking information for specifying a criminal as a tracking object and tracking the object if the risk situation is a crime. In an embodiment, the tracking information can be transmitted to relevant organizations such as police officers around the tracking object, an administrator terminal, and an administrator terminal included in the expected route.

[0072] In addition, in the embodiment, the event visualization unit (150) generates and outputs traffic information based on the results of CCTV image analysis of roads and passageways. In the embodiment, the traffic information includes information on the degree of traffic congestion on each road, vehicle distribution, and whether or not an accident has occurred. Thereafter, the event visualization unit (150) can extract roads with traffic congestion information above a certain level, calculate the cause of the traffic congestion and the expected time for resolution, and transmit the information to the manager terminal where the traffic congestion occurs. In addition, if assistance from a rescue agency is required after a dangerous situation occurs, the event visualization unit (150) can extract an optimal movement route to the location of the dangerous situation based on traffic information and transmit the information to the rescue agency.

[0073] The feedback unit (160) evaluates the learned artificial neural network model and deep learning model. In an embodiment, the feedback unit (160) can evaluate the artificial neural network model through at least one of accuracy, precision, and recall. Accuracy is an index that measures how much the results predicted by the artificial neural network model match the actual results. Precision is an index that measures the ratio of actual positives among the results predicted as positive. Recall is an index that measures the ratio of actual positives predicted by the model as positives. In an embodiment, the feedback unit (160) can calculate the accuracy, precision, and recall of the artificial neural network model, and evaluate the artificial neural network model based on at least one of the calculated indexes.

[0074] In an embodiment, the feedback unit (160) can measure the accuracy of an artificial neural network model using an evaluation dataset. The evaluation dataset consists of data that the model did not use for training and is used to objectively evaluate the model's performance. In an embodiment, the feedback unit (160) executes the artificial neural network model using the evaluation dataset and compares the predicted value of the artificial neural network model for each input data with the actual correct answer value of the corresponding data. Thereafter, the accuracy of the model's predictions can be measured based on the comparison results. For example, the accuracy in the feedback unit (160) can be calculated as the ratio of data correctly predicted by the model among the entire data.

[0075] In addition, the feedback unit (160) can calculate the F1 score, which is an index indicating the balance of precision and recall, which is an index calculated as the harmonic mean of precision and recall, evaluate the artificial neural network model based on the calculated F1 score, generate an AUC-ROC curve, which is an index that visualizes the performance of the classification model in a graph, and evaluate the artificial neural network model based on the generated AUC-ROC curve. In an embodiment, the feedback unit (160) can evaluate that the performance of the model is better as the area under the ROC curve (AUC) is closer to 1.

[0076] In addition, the feedback unit (160) can evaluate the interpretability of the artificial neural network model. In an embodiment, the feedback unit (160) evaluates the interpretability of the artificial neural network model through SHAP (SHapley Additive exPlanations) and LIME (Local Interpretable Model-agnostic Explanations) methods. SHAP (SHapley Additive exPlanations) is a library that provides an interpretation of the results predicted by the model, and the feedback unit (160) extracts SHAP values ​​from the library. In an embodiment, the feedback unit (160) can predict how much the characteristic information input to the model influenced the model prediction through the SHAP value extraction.

[0077] A model in this specification may refer to any form of computer program that operates based on a network function, an artificial neural network, and / or a neural network. Throughout this specification, the terms model, neural network, network function, and neural network may be used interchangeably. A neural network is a network in which one or more nodes are interconnected through one or more links to form input node and output node relationships within the neural network. The characteristics of a neural network can be determined based on the number of nodes and links within the neural network, the correlation between the nodes and links, and the weight value assigned to each link. A neural network may be composed of a set of one or more nodes. A subset of the nodes constituting the neural network may constitute a layer.

[0078] A deep neural network (DNN) may refer to a neural network that includes multiple hidden layers in addition to an input layer and an output layer. A deep neural network may include a convolutional neural network (CNN), a recurrent neural network (RNN), an autoencoder, a generative adversarial network (GAN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a Q network, a U network, a Siamese network, a generative adversarial network (GAN), a transformer, and the like. The description of the above-described deep neural network is merely an example, and the present disclosure is not limited thereto.

[0079] Neural networks can learn through at least one of the following methods: supervised learning, unsupervised learning, semi-supervised learning, self-supervised learning, or reinforcement learning. Neural network learning can be the process of applying knowledge to the neural network to perform a specific action.

[0080] Neural networks can be trained to minimize output errors. This process involves repeatedly inputting training data into the neural network, calculating the neural network output and target error for the training data, and backpropagating the neural network error from the output layer to the input layer to update the weights of each node in the neural network to reduce the error. In supervised learning, labeled data is used for each training data, while unsupervised learning uses unlabeled data. The amount of change in the connection weights of each updated node can be determined by the learning rate. The neural network's calculation of input data and backpropagation of errors can constitute a learning cycle (epoch). The learning rate can vary depending on the number of iterations in the neural network's training cycle. Additionally, to prevent overfitting, methods such as increasing the learning data, regularization, dropout that disables some nodes, and batch normalization layers can be applied.

[0081] In one embodiment, the model may include, but is not limited to, at least one of a Recurrent Neural Network (RNN), a Long Short Term Memory (LSTM) network, a Deep Neural Network (DNN), a Convolutional Neural Network (CNN), and a Bidirectional Recurrent Deep Neural Network (BRDNN).

[0082] In one embodiment, the model may be a model trained using transfer learning. Transfer learning, in this context, refers to a learning method that pre-trains a large amount of unlabeled training data using semi-supervised or self-learning methods to obtain a pre-trained model for a first task, then fine-tunes the pre-trained model to suit a second task, and trains it on labeled training data using supervised learning to implement a target model.

[0083] The GIS AI CCTV map-based data visualization urban operations management system and server described above provides a data-driven control service that leverages the intuitive nature of GIS to quickly identify the cause, track issues, and initiate early response measures. This allows city managers to immediately recognize problems that arise, accurately identify the cause based on location information, and take swift action.

[0084] In addition, the GIS artificial intelligence CCTV map-based data visualization urban operation management system and server according to the embodiment improves the efficiency of data-based urban management by providing objective data on the area where a problem occurred or the problem that occurred along with the video.

[0085] The disclosed content is merely an example, and various modifications and implementations can be made by a person skilled in the art without departing from the gist of the claims claimed in the patent, so the scope of protection of the disclosed content is not limited to the specific embodiments described above.

Claims

1. In the GIS artificial intelligence CCTV map-based data visualization urban operation management system, CCTV that generates monitoring images for each area of ​​the city; IOT sensors that sense urban data, which is data on urban elements including transportation, environment, population, and energy; A server that collects city data from the IOT sensor, detects dangerous situations through object recognition included in images collected from the CCTV, extracts related data on the detected dangerous situations, and visualizes the CCTV image analysis results; and An urban operation management system including an administrator terminal that outputs the CCTV video analysis results of the above server and the visualization results of urban data.

2. In paragraph 1, The above server; is, Data collection unit that collects real-time video data and learning data from CCTV; A metadata generation unit that constructs big data by auto-labeling collected image data, trains an object recognition model with the big data, preprocesses unstructured data, which is urban data, and constructs metadata by fusing the preprocessed unstructured data and the image data processing results; A metadata learning unit that sets a threshold according to urban management rules using the above-mentioned metadata and creates a problem situation event detection model based on a rule engine; A problem situation event detection unit that detects a problem situation event by inputting real-time metadata into the above-generated problem situation event detection model; and An urban operation management system characterized by including an event visualization unit that performs visualization of GIS map linkage according to a detected problem situation and performs map replay according to the problem situation.

3. In the second paragraph, the metadata generation unit; Collect CCTV video data and city data, and perform auto-labeling through learning on the collected data to convert unstructured data into metadata. Processing image data applied to the above metadata by applying an artificial intelligence-based object recognition algorithm, The above object recognition algorithm An urban operation management system characterized by being based on at least one of a facial recognition algorithm, an object recognition algorithm, and a crowd behavior recognition algorithm.

4. In paragraph 2, The above metadata learning unit, Based on the constructed metadata, a problem situation event detection model is constructed by setting thresholds corresponding to the caution stage and emergency stage to detect problem situation events according to the urban management rules. The above problem situation event detection unit is, An urban operation management system characterized in that it applies CCTV footage and city data input in real time to the above-mentioned problem situation event detection model to calculate a rule violation value, and detects a problem situation event when the calculated rule violation value exceeds a caution stage threshold.

5. In paragraph 4, The above problem situation event detection unit is, An urban operation management system characterized by collecting CCTV footage and city data as real-time data, inputting the real-time data into the problem situation event detection model to detect problem situation events in real time, and transmitting the detection results to an administrator terminal.

6. In paragraph 5, The above problem situation event detection unit is, Input metadata into the problem situation event detection model to calculate the rule violation number, If the calculated rule violation value exceeds the preset caution level threshold, it is detected as a problem situation event. An urban operation management system characterized in that, when the calculated rule violation number exceeds a preset emergency level threshold, an alarm is transmitted to a preset administrator terminal and a public institution terminal or server.

7. In paragraph 2, The above event visualization section, An urban operation management system characterized in that, when a problem situation event occurs, CCTV footage and urban data linked to the problem situation event are fused to visualize location-based data linked to a GIS map.

8. In paragraph 7, The above event visualization section, An urban operation management system characterized in that, when a user terminal or administrator terminal designates a specific point in time at which a problem situation occurs, the system replays the situation before and after the problem situation occurs by fusing video data and city data of a location linked to the designated problem situation.

9. In paragraph 7, The above event visualization section, An urban operation management system characterized in that when a problem situation event is detected, metadata related to the problem situation event is input into a risk classification model that classifies the risk of the problem situation event, and if the risk falls within a preset risk level, all adjacent CCTVs in the area where the problem situation event occurred are called and additional CCTV footage is collected from the called CCTVs.

10. In paragraph 9, The above additional CCTV footage is: An urban operation management system characterized by including tracking information for tracking an object by specifying the criminal as a tracking object when the dangerous situation is a crime.

11. In paragraph 10, The above event visualization section, When information on a tracking object to be detected is input, tracking information is generated by comparing the tracking object information with the facial area extracted from the CCTV image. An urban operation management system characterized in that it extracts the expected location of a tracked object at the next time through tracking information, derives an expected route including the expected location, calls a CCTV adjacent to the derived expected route to set the tracked object to be tracked in advance, and acquires additional information from a CCTV adjacent to the expected route.

12. In the GIS artificial intelligence CCTV map-based data visualization city operation management server, Collect city data from IoT sensors, detect dangerous situations through object recognition included in the images collected from the CCTV, extract related data on the detected dangerous situations, and visualize the CCTV image analysis results. The above server; Data collection unit that collects real-time video data and learning data from CCTV; A metadata generation unit that constructs big data by auto-labeling collected image data, trains an object recognition model with the big data, preprocesses unstructured data, which is urban data, and constructs metadata by fusing the preprocessed unstructured data and the image data processing results; A metadata learning unit that sets a threshold according to urban management rules using the above-mentioned metadata and creates a problem situation event detection model based on a rule engine; A problem situation event detection unit that detects a problem situation event by inputting real-time metadata into the above-generated problem situation event detection model; and A city operation management server including an event visualization unit that performs visualization of GIS map linkage according to a detected problem situation and performs map replay according to the problem situation.

13. In paragraph 12, The above metadata generation unit. Collect CCTV video data and city data, and perform auto-labeling through learning on the collected data to convert unstructured data into metadata. Processing image data applied to the above metadata by applying an artificial intelligence-based object recognition algorithm, The above object recognition algorithm is, A city operation management server characterized by being based on at least one of a facial recognition algorithm, an object recognition algorithm, and a crowd behavior recognition algorithm.

14. In paragraph 12, The above metadata learning unit, Based on the constructed metadata, a problem situation event detection model is constructed by setting thresholds corresponding to the caution stage and emergency stage to detect problem situation events according to the urban management rules. The above problem situation event detection unit is, Input metadata into the problem situation event detection model to calculate the rule violation number, If the calculated rule violation value exceeds the preset caution level threshold, it is detected as a problem situation event. A city operation management server characterized in that, when the calculated rule violation number exceeds a preset emergency level threshold, an alarm is transmitted to a preset administrator terminal and a public institution terminal or server.

15. In paragraph 12, The above event visualization section, When a problem situation event occurs, CCTV footage and city data linked to the problem situation event are merged to visualize location-based data linked to a GIS map. A city operation management server characterized in that, when a user terminal or administrator terminal designates a specific point in time at which a problem situation occurs, it replays the situation before and after the problem situation occurs by fusing video data and city data of a location linked to the designated problem situation.

Images