Situation information visual analysis method and device, electronic equipment and storage medium

Abstract

The invention relates to a situation information visual analysis method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a data source through a calculation engine by monitoring data in a message consumption queue, carrying out the cleaning preprocessing of the data source, and storing the cleaned and preprocessed data source in a database. And calling an algorithm platform to carry out calculation and deep analysis on the data source in the database to obtain a situation information portrait, and storing the situation information portrait in a data portrait information database so as to call the situation information portrait in real time and map the situation information portrait to an information visual layer. The data source comprises multi-type comprehensive situation data of multiple sources, the comprehensive situation data comprises structured data and unstructured data, the algorithm platform is constructed based on multiple basic algorithms and a machine learning framework, and the basic algorithms at least comprise a text mining algorithm, a classification algorithm, a regression algorithm, a clustering algorithm and a recommendation algorithm. And the analysis comprehensiveness, accuracy and visual effect of the multi-source situation data are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of comprehensive situation awareness, and in particular to a situation information visual analysis method, device, electronic equipment and storage medium. Background Art

[0002] In the field of comprehensive situational awareness, multi-source situational information is displayed in all dimensions from different angles and levels, providing support for a comprehensive and timely grasp of the current status of the space environment domain. It is the decision-making basis for typical tasks such as mission planning and program implementation, and an important guarantee for the precise realization of tasks. With the rapid development of informatization, comprehensive situational awareness has been elevated to a strategic level and has become a hot topic in the field of comprehensive information display. In addition, comprehensive situation is an organic combination of multiple situational information. On the basis of standardized processing of multi-source situational data resources and orderly mining to form situational information, a certain organizational management method can be used to ensure the integrity and accuracy of situational information in the process of processing, analysis, and display.

[0003] At present, under the flood of data in the big data era, the types of situation data are becoming more and more abundant, and the amount of data is also increasing. The data has obvious big data characteristics, such as large capacity, multiple types, difficult to identify, and high value. With the deepening integration of business and information technology, the problem faced by comprehensive situation awareness technology is no longer the lack of data, but how to extract value from massive data and make the data valuable. However, traditional comprehensive situation awareness technology is often affected by the massive types of situation data, and the analysis of situation data is relatively incomplete, the analysis accuracy is also low, and the visual effect is not intuitive enough. Summary of the invention

[0004] Based on this, it is necessary to provide a situation information visual analysis method, device, electronic device and storage medium that can improve the comprehensiveness, accuracy and visual intuitive effect of analysis of various types of situation data from various sources in response to the above technical problems.

[0005] The present invention provides a situation information visual analysis method, the method comprising: The computing engine obtains a data source by monitoring data in a message consumption queue, and cleans and pre-processes the data source, so as to store the cleaned and pre-processed data source in a database; Calling the algorithm platform to calculate and deeply analyze the data source in the database to obtain a situation information portrait, and storing the situation information portrait in the data portrait information database, so as to retrieve the situation information portrait in real time and map it to the information visualization layer; Among them, the data source includes multi-type comprehensive situation data from multiple sources, the comprehensive situation data includes structured data and unstructured data, the algorithm platform is constructed based on multiple basic algorithms and machine learning frameworks, and the basic algorithms include at least text mining algorithms, classification algorithms, regression algorithms, clustering algorithms and recommendation algorithms.

[0006] In one embodiment, the obtaining of the data source by the computing engine by monitoring the data in the message consumption queue, and cleaning and preprocessing the data source to store the cleaned and preprocessed data source in the database includes: Extracting the unstructured data from the data source, and performing data quality filtering on the unstructured data through a defined filtering model or filtering rule, so as to filter out low-quality data in the unstructured data whose data quality does not meet set requirements; The Bloom filter technology is called to perform data deduplication on the unstructured data after filtering to obtain the preprocessed unstructured data, and the structured data and the preprocessed unstructured data are stored in the database.

[0007] In one embodiment, the calling algorithm platform calculates and deeply analyzes the data source in the database to obtain a situation information portrait, and stores the situation information portrait in a data portrait information database, so as to retrieve the situation information portrait in real time and map it to the information visualization layer, which includes: The algorithm platform is constructed based on the text mining algorithm, classification algorithm, regression algorithm, clustering algorithm and recommendation algorithm combined with the machine learning framework Flink; The algorithm platform is used to abstract the calculation and in-depth analysis algorithms corresponding to the multiple types of comprehensive situation data from the multiple sources, so as to optimize the algorithm platform and obtain the calculation framework of each type of the comprehensive situation data.

[0008] In one embodiment, the calling algorithm platform calculates and deeply analyzes the data source in the database to obtain a situation information portrait, and stores the situation information portrait in a data portrait information database, so as to retrieve the situation information portrait in real time and map it to the information visualization layer, including: Calling the algorithm platform to establish an analysis and processing model corresponding to each type of the comprehensive situation data according to different types of the comprehensive situation data; Fitting a plurality of processing and analysis tasks corresponding to the same type of the comprehensive situation data through the analysis and processing model to obtain processing and analysis results; Among them, data is shared between the same type of comprehensive situation data, and each processing and analysis task corresponding to the same type of comprehensive situation data has shared parameters, and the processing and analysis results are obtained by fusing the analysis result outputs of multiple subtasks of each processing and analysis task.

[0009] In one embodiment, the calling algorithm platform calculates and deeply analyzes the data source in the database to obtain a situation information portrait, and stores the situation information portrait in a data portrait information database to retrieve the situation information portrait in real time and map it to the information visualization layer, and also includes: Determine the fusion weight of each subtask in the processing and analysis task according to the data discreteness and data scale of the corresponding type of the comprehensive situation data, and call the fusion formula to fuse the analysis result outputs of multiple subtasks according to the fusion weight of each subtask; The calculation formula of the fusion weight is: ; In the formula, is the data discreteness of the comprehensive situation data, is the data scale of the comprehensive situation data, is the fusion weight; The fusion formula is: ; In the formula, result is the total score after fusion, n is the number of subtasks, and result i is the analysis result output of the i-th subtask, is the fusion weight of the i-th subtask.

[0010] In one embodiment, the calling algorithm platform calculates and deeply analyzes the data source in the database to obtain a situation information portrait, and stores the situation information portrait in a data portrait information database to retrieve the situation information portrait in real time and map it to the information visualization layer, and also includes: Acquire the comprehensive situation data from multiple sources, and divide the comprehensive situation data into offline data and real-time data; Calling the offline processing layer and the near-line analysis layer in the algorithm platform to process and analyze the offline data and the real-time data, and calling the online computing layer to perform calculation and in-depth analysis on the processed and analyzed offline data and the real-time data to generate the situation information portrait; The algorithm platform includes the offline processing layer, the near-line analysis layer and the online calculation layer.

[0011] In one of the embodiments, the algorithm platform further includes the information visualization layer; The calling algorithm platform calculates and deeply analyzes the data source in the database to obtain a situation information portrait, and stores the situation information portrait in a data portrait information database to retrieve the situation information portrait in real time and map it to the information visual layer, and also includes: The situation information portrait generated by the online computing layer is retrieved in real time from the data portrait information database and sent to the information visualization layer of the algorithm platform to map the situation information portrait to the information visualization layer.

[0012] The present invention also provides a situation information visual analysis device, the device comprising: A data preprocessing module is used to obtain a data source by monitoring the data in the message consumption queue through a computing engine, and to clean and preprocess the data source to store the cleaned and preprocessed data source in a database; An information visual analysis module is used to call the algorithm platform to perform calculation and in-depth analysis on the data source in the database to obtain a situation information portrait, and store the situation information portrait in a data portrait information database so as to retrieve the situation information portrait in real time and map it to the information visual layer; Among them, the data source includes multi-type comprehensive situation data from multiple sources, the comprehensive situation data includes structured data and unstructured data, the algorithm platform is constructed based on multiple basic algorithms and machine learning frameworks, and the basic algorithms include at least text mining algorithms, classification algorithms, regression algorithms, clustering algorithms and recommendation algorithms.

[0013] The present invention also provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements any one of the above-mentioned situation information visual analysis methods when executing the computer program.

[0014] The present invention also provides a computer storage medium storing a computer program, wherein when the computer program is executed by a processor, the situation information visual analysis method described in any one of the above is implemented.

[0015] The above-mentioned situation information visual analysis method, device, electronic device and storage medium, by establishing integrated management of various types of situation data resources and using big data processing and visualization analysis technology for situation awareness, can more comprehensively analyze different types of multi-source situation information, and then give full play to the advantages of technologies such as intelligent detection and big data in-depth analysis, form comprehensive situation information with complete elements and rich content, organically combine the analysis results of multiple types of situation data, and finally form a visualization view of the comprehensive situation information, which can achieve a comprehensive grasp and precise control of the spatial environment situation, and improve the comprehensiveness of the analysis of multiple types of situation data from multiple sources, the accuracy of the analysis and the intuitiveness of the visual effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 This is one of the flow charts of the situation information visual analysis method provided by the present invention; Figure 2 The second flowchart of the situation information visual analysis method provided by the present invention; Figure 3 The third flowchart of the situation information visual analysis method provided by the present invention; Figure 4 The fourth flowchart of the situation information visual analysis method provided by the present invention; Figure 5 The fifth flow chart of the situation information visual analysis method provided by the present invention; Figure 6 A schematic diagram of the overall architecture of a comprehensive situation information visual analysis system of a situation information visual analysis method in a specific embodiment provided by the present invention; Figure 7 A schematic diagram of a framework flow of a comprehensive situation information visual analysis system of a situation information visual analysis method in a specific embodiment provided by the present invention; Figure 8 A schematic diagram of a comprehensive situation data preprocessing process of a situation information visual analysis method in a specific embodiment provided by the present invention; Fig. 9 A schematic diagram of a process of constructing a portrait based on a comprehensive situation data set in a situation information visual analysis method in a specific embodiment provided by the present invention; Fig.10A schematic diagram of a multi-task data processing analysis model structure of a situation information visual analysis method in a specific embodiment provided by the present invention; Fig.11 A schematic diagram of the flow of situation data processing and analysis of visual information of the situation information visual analysis method in a specific embodiment provided by the present invention; Fig.12 A schematic diagram of the structure of the situation information visual analysis device provided by the present invention; Fig.13 This is a diagram of the internal structure of the computer device provided by the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] Combine the following Figure 1-Figure 13 The present invention describes the situation information visual analysis method, device, electronic device and storage medium.

[0020] like Figure 1 As shown, in one embodiment, a situation information visual analysis method includes the following steps: Step S110, the computing engine obtains the data source by monitoring the data in the message consumption queue, and cleans and pre-processes the data source to store the cleaned and pre-processed data source in the database.

[0021] Specifically, the server obtains the data source by monitoring the data in the message consumption queue through the computing engine, and cleans and preprocesses the data source to store the cleaned and preprocessed data source in the database, where the data source includes multi-type comprehensive situation data from multiple sources, and the comprehensive situation data includes structured data and unstructured data.

[0022] Step S120, calling the algorithm platform to perform calculations and in-depth analysis on the data sources in the database to obtain a situation information portrait, and storing the situation information portrait in a data portrait information database, so as to retrieve the situation information portrait in real time and map it to the information visualization layer.

[0023] Among them, the algorithm platform is built based on a variety of basic algorithms and machine learning frameworks. The basic algorithms include but are not limited to text mining algorithms, classification algorithms, regression algorithms, clustering algorithms and recommendation algorithms.

[0024] Specifically, the server calls an algorithm platform built based on a variety of basic algorithms and machine learning frameworks to calculate and deeply analyze the data sources in the database, obtain a situation information portrait, and store the situation information portrait in a data portrait information database, so as to retrieve the situation information portrait in real time and map it to the information visualization layer.

[0025] The above-mentioned situation information visual analysis method, by establishing integrated management of various types of situation data resources and using big data processing and visualization analysis technology for situation awareness, can more comprehensively analyze different types of multi-source situation information, and then give full play to the advantages of technologies such as intelligent detection and big data in-depth analysis, to form comprehensive situation information with complete elements and rich content, organically combine the analysis results of multiple types of situation data, and finally form a visualization view of the comprehensive situation information, which can achieve a comprehensive grasp and precise control of the spatial environment situation, and improve the comprehensiveness of the analysis of multiple types of situation data from multiple sources, the accuracy of the analysis and the intuitiveness of the visual effects.

[0026] like Figure 2 As shown, in one embodiment, the situation information visual analysis method provided by the present invention obtains a data source by monitoring the data in the message consumption queue through a computing engine, and cleans and pre-processes the data source to store the cleaned and pre-processed data source in a database, specifically comprising the following steps: Step S112, extracting unstructured data from the data source, and performing data quality filtering on the unstructured data through a defined filtering model or filtering rule, so as to filter out low-quality data in the unstructured data whose data quality does not meet the set requirements.

[0027] Specifically, the server extracts unstructured data from the data source, and performs data quality filtering on the unstructured data through a defined filtering model or filtering rules to filter out low-quality data in the unstructured data whose data quality does not meet the set requirements.

[0028] Step S114, calling the Bloom filter technology to perform data deduplication on the filtered unstructured data to obtain the preprocessed unstructured data, and storing the structured data and the preprocessed unstructured data in a database.

[0029] Specifically, the server calls the Bloom filter technology to deduplicate the unstructured data after filtering, obtains the preprocessed unstructured data, and stores the structured data and the preprocessed unstructured data in a database.

[0030] Among them, Bloom Filter is a very space-saving probabilistic data structure with fast running speed and small memory usage. It is actually composed of a very long binary vector and a series of random mapping functions, and is mainly used to determine whether an element is in a set.

[0031] like Figure 3 As shown, in one embodiment, the situation information visual analysis method provided by the present invention calls the algorithm platform to calculate and deeply analyze the data source in the database to obtain the situation information portrait, and stores the situation information portrait in the data portrait information database to retrieve the situation information portrait in real time and map it to the information visual layer, which includes the following steps: Step S310, building an algorithm platform based on text mining algorithm, classification algorithm, regression algorithm, clustering algorithm and recommendation algorithm combined with machine learning framework Flink.

[0032] Specifically, the server builds an algorithm platform based on text mining algorithm, classification algorithm, regression algorithm, clustering algorithm and recommendation algorithm combined with the machine learning framework Flink.

[0033] Among them, Flink is a unified computing framework that combines batch processing and stream processing. Its core is a stream data processing engine that provides data distribution and parallel computing.

[0034] Step S320, abstracting the calculation and in-depth analysis algorithms corresponding to multiple types of comprehensive situation data from multiple sources through the algorithm platform, so as to optimize the algorithm platform and obtain the calculation framework of each type of comprehensive situation data.

[0035] Specifically, the server abstracts the calculation and in-depth analysis algorithms corresponding to multiple types of comprehensive situation data from multiple sources through the algorithm platform constructed in step S310, so as to optimize the algorithm platform and obtain the calculation framework of each type of comprehensive situation data.

[0036] like Figure 4 As shown, in one embodiment, the situation information visual analysis method provided by the present invention calls the algorithm platform to calculate and deeply analyze the data source in the database to obtain the situation information portrait, and stores the situation information portrait in the data portrait information database to retrieve the situation information portrait in real time and map it to the information visual layer, which specifically includes the following steps: Step S121, calling the algorithm platform to establish an analysis and processing model corresponding to each type of comprehensive situation data according to different types of comprehensive situation data.

[0037] Specifically, the server calls the algorithm platform to establish an analysis and processing model corresponding to each type of comprehensive situation data according to different types of comprehensive situation data.

[0038] Step S123, fitting multiple processing and analysis tasks corresponding to the same type of comprehensive situation data through the analysis and processing model to obtain processing and analysis results.

[0039] Specifically, the server fits multiple processing and analysis tasks corresponding to the same type of comprehensive situation data through the analysis and processing model to obtain the processing and analysis results.

[0040] It should be noted that in the process of multi-task analysis and processing, the server determines the fusion weight of each subtask in the processing and analysis task according to the data discreteness and data scale of the corresponding type of comprehensive situation data, and calls the fusion formula to fuse the analysis result outputs of multiple subtasks according to the fusion weight of each subtask.

[0041] The calculation formula of the fusion weight is: ; In the formula, is the data discreteness of the comprehensive situation data, is the data scale of the comprehensive situation data, is the fusion weight.

[0042] The fusion formula is: ; In the formula, result is the total score after fusion, n is the number of subtasks, and result i is the analysis result output of the i-th subtask, is the fusion weight of the i-th subtask.

[0043] In addition, data is shared between the same type of comprehensive situation data, and each processing and analysis task corresponding to the same type of comprehensive situation data has shared parameters, and the processing and analysis results are obtained by fusing the analysis result outputs of multiple subtasks of each processing and analysis task.

[0044] like Figure 5 As shown, in one embodiment, the situation information visual analysis method provided by the present invention calls the algorithm platform to calculate and deeply analyze the data source in the database to obtain the situation information portrait, and stores the situation information portrait in the data portrait information database to retrieve the situation information portrait in real time and map it to the information visual layer, and specifically includes the following steps: Step S122, obtaining comprehensive situation data from multiple sources, and dividing the comprehensive situation data into offline data and real-time data.

[0045] Specifically, the server obtains comprehensive situation data from multiple sources, and divides the comprehensive situation data into offline data and real-time data.

[0046] Step S124, calling the offline processing layer and near-line analysis layer in the algorithm platform to process and analyze the offline data and real-time data, and calling the online computing layer to perform calculations and in-depth analysis on the processed and analyzed offline data and real-time data to generate a situation information portrait.

[0047] Specifically, the server calls the offline processing layer and near-line analysis layer in the algorithm platform to process and analyze offline data and real-time data, and calls the online computing layer to perform calculations and in-depth analysis on the processed and analyzed offline data and real-time data to generate a situation information portrait.

[0048] Step S126, retrieve the situation information portrait generated by the online computing layer from the data portrait information database in real time, and send it to the information visualization layer of the algorithm platform to map the situation information portrait to the information visualization layer.

[0049] Specifically, the server retrieves the situation information portrait generated by the online computing layer from the previously obtained data portrait information database in real time, and sends it to the information visualization layer of the algorithm platform to map the situation information portrait to the information visualization layer.

[0050] See also Figure 6 As shown, in a specific embodiment, the present invention provides a situation information visual analysis method, which is mainly implemented by a comprehensive situation information visual analysis system. The comprehensive situation information visual analysis system is a highly comprehensive system, and the system architecture adopts a four-layer architecture, including a support environment, a data processing and analysis layer, a service layer, and a comprehensive situation portrait visualization layer. Through a multi-level design, the processing flow of specific functions is deposited in each layer.

[0051] Among them, the support environment provides basic software and hardware resources, the data processing and analysis layer includes two modules: situation information big data evaluation and analysis and situation information portrait generation, and the service layer provides services such as human situation services, material situation services, equipment and facility situation services, network situation services, data situation services, and task situation services. The comprehensive situation portrait visualization layer provides functions such as human situation visualization, material situation visualization, equipment and facility situation visualization, network situation visualization, data situation visualization, and task situation visualization.

[0052] In this embodiment, the support environment is used to provide the servers required for big data services, the databases of structured, semi-structured and unstructured data, and the software and hardware resource access systems to achieve access support for massive data. The data processing and analysis layer is used to comprehensively perceive and analyze the situation in terms of human resources, materials, equipment and facilities, networks, data, tasks, etc., as well as evaluate and predict. The data processing and analysis layer, with the core around these data contents, collects comprehensive situation-related data from various data sources in real time, provides the ability to accept various types of data based on the big data system, and builds data storage in the big data processing platform, laying the foundation for analyzing and processing massive, multimodal data. The situation information big data evaluation and analysis module is used to provide analysis and evaluation prediction capabilities for various situation information. The full life cycle of data includes data collection, data processing, data transmission, data storage and use. Data collection and processing is the basis for data analysis and evaluation. Only based on objective, real and comprehensive data can effective and accurate situation information results be analyzed. In order to ensure the availability and high quality of the collected data and meet the analysis objectives, situation data can be divided into two types: static data and dynamic data. Static information data mainly comes from user filling and system-level acquisition. Because the collected static information is uncertain, it will be modeled, judged and improved. Dynamic information data is hidden, referring to the constantly changing data with big data characteristics and real-time characteristics, which needs to be accurately extracted through data analysis and data mining.

[0053] After collecting static and dynamic information of situation data, on the premise of fully protecting data privacy and ensuring the authenticity and validity of the data, first filter out redundant data and abnormal information that are not related to the characteristics of each situation data; then process the cleaned situation data to make it data that can be used for modeling various situation information such as network, human resources, and data.

[0054] The situation information portrait generation module is responsible for generating situation portraits of various situation data based on the results data generated by the evaluation and analysis module. The portraits can directly and indirectly reflect the data capabilities. Combining static and dynamic information such as network, manpower, materials, equipment and facilities, and data, relatively three-dimensional and accurate network portrait models, manpower portrait models, material portrait models, equipment and facilities portrait models, and data portrait models are constructed respectively. With reference to existing factual data, the situation information is evaluated through algorithms to obtain prediction information and explore potential capabilities.

[0055] In this embodiment, the service layer is based on the well-established situation information portrait model, and various service engines are designed in combination with specific businesses, such as equipment and facility situation query engines, network situation query engines, etc., which are responsible for obtaining portrait data after real-time calculation and analysis to support various applications of the comprehensive situation portrait visualization layer based on situation portraits. The comprehensive situation portrait visualization layer is used to meet the complex data exploration and mining needs of the comprehensive situation. By developing a system for visualizing comprehensive situation information, it helps users implement more complete data analysis in the field of comprehensive situation awareness. The comprehensive situation portrait visualization layer combines an open source big data visualization engine to visualize the knowledge-based comprehensive situation information after analysis and processing, and uses relatively simple visual graphics to convey very high-density situation information, helping users better understand the patterns, laws or anomalies in the data, and make judgments and decisions.

[0056] The comprehensive situation portrait visualization layer uses a customized WebGL-based geographic information platform to implement a big data visualization analysis framework base with geographic space. It realizes map-related capabilities by connecting to the platform engine and framework, and realizes the ability to overlay various visual symbols on the map to undertake different analysis and presentation tasks. First, the map base map is connected through the engine to express geographic information. Then, the map is overlaid with the situation positioning data set to accurately draw the focus points on the precise map, so as to emphasize the most critical geographic location information of the focus points, and realize the visualization analysis function combining the focus points with geography.

[0057] In addition, the comprehensive situation portrait visualization layer obtains situation information data based on the data portraits and evaluation results of various situation information such as manpower, materials, equipment and facilities, networks, data, and tasks by connecting to various services in the service layer, and performs intuitive data visualization on demand, integrating it into comprehensive situation information visualization empowerment, and finally applying it in reality, allowing users to quickly and accurately understand the true form of the data, while discovering the value in the data and assisting in data analysis and data mining.

[0058] It should be noted that the content of comprehensive situation data analysis is multi-dimensional and relatively complex. Understanding comprehensive situation data through appropriate visualization means is an important goal of the comprehensive situation information visual analysis system. Common data visualization relationship categories such as comparison, trend, proportion, distribution, correlation, hierarchy, and association are not enough to fully demonstrate the purpose of multi-source and multi-category situation data. Therefore, it is necessary to follow Web standards, use SVG, Canvas, and HTML to turn data into reality, combine powerful visualization and interaction technologies with data-driven DOM operation methods, and design composite charts such as the combination of stacked column charts and line charts, and the combination of tables and timelines, so as to freely design appropriate visualization interfaces for multi-source and multi-category situation data. Finally, the situation of manpower, network, equipment and facilities, materials, tasks, etc. is outlined with a simple view, and the intuitive data presentation assists users to analyze a wide variety of data, giving full play to the advantages of visualization in assisting perception.

[0059] Combination Figure 7 As shown, in this embodiment, the comprehensive situation information visual analysis system aims to dig out the truth and discover the law from the existing situation data as the basis for behavior and judgment. The comprehensive situation information visual analysis system framework integrates data collection, data storage and management, data analysis and calculation, data visualization, and data security and privacy. It has accurate data statistical analysis and data mining functions, and the conclusions usually have certain decision-making.

[0060] First, the real-time computing engine collects objective and rational data by monitoring the data in the message consumption queue, and writes the collected data sources into the database in a unified data format for storage. After collecting enough information, the data in the database is calculated and analyzed in real time by running natural language processing, classification and clustering components, and the situation information portrait content is obtained in a more real-time and accurate manner, realizing real-time calculation of massive data. The result after calculation is often factual information. After a series of deep reasoning and integration, the situation data resources are sublimated into knowledge and stored in the data portrait information database for real-time retrieval and use, providing support for various precision services. Finally, the data is mapped to the visual layer, and graphs are used to express the information contained in the data.

[0061] In the comprehensive situation information visual analysis system, a large amount of comprehensive situation data set resources are introduced. A large amount of data can improve the accuracy of mining and analysis, but the unstructured texts from a wide range of sources are often mixed with a lot of low-quality data. Data quality is the basis of big data visual analysis. Ensuring data quality is the key to obtaining accurate and reliable analysis results. Before the collected data is analyzed, it must be carefully preprocessed and cleaned. In order to improve data quality, optimization has been carried out in terms of data sources, cleaning rules, storage and backup, and appropriate data verification and validation methods have been adopted.

[0062] Specifically, collect multi-type comprehensive situation data from various sources and land them on databases or distributed file systems. Unstructured data needs to be cleaned through pre-processing processes such as quality filtering and content deduplication to purify the data set, realize the transformation from unstructured to structured, standardize the information and store it in HBase and relational databases, make the data more readable, and mine its value offline or online in the next step. Data quality filtering is the key to data preprocessing, and it can be filtered according to defined rules or models. In many cases, the data content is the same, and Bloom Filter technology is used to quickly deduplicate the data, collect and form an effective data set, and accelerate data processing efficiency.

[0063] Combination Figure 8 As shown, in this embodiment, due to the rich types of situation data and different characteristics, there are differences in processing logic. An algorithm platform is constructed based on basic algorithms such as text mining algorithms, classification algorithms, regression algorithms, clustering algorithms, recommendation algorithms, and technical means such as the machine learning framework Flink. In response to different data situations, the algorithm platform abstracts more detailed algorithm methods for specific business data processing, realizes fine mining of specific types of data, and continuously optimizes the algorithm until effective information is mined. Each computing framework mines and generates effective data such as data portraits and knowledge, exchanges them through the intermediate medium of NoSQL, and quickly provides services to the outside through the online service engine, reflecting the value of the data and ultimately achieving the goal of data mining.

[0064] Combination Fig. 9 As shown, in this embodiment, the combination of big data technology and data profiling improves the accuracy of profiling while greatly increasing the scale of processable data, and can provide more refined and dimensionally rich visualization information. Based on different types of data sets, corresponding data analysis and processing models are established. The same type of data is shared through data sharing, and multiple processing and analysis tasks are simultaneously fitted using one model. Some parameters are shared between tasks, which reduces the overall size of the model, reduces memory overhead, saves resources, and improves the generalization and expression capabilities of the model.

[0065] Combination Fig.10 As shown, in this embodiment, in the multi-task form of data processing and analysis, each subtask has an analysis result output, and their results need to be fused as the final processing and analysis result. For analysis tasks related to scoring and recommendation, such as data contribution, ranking, and sorting, it is necessary to fuse their score results. First, the fusion weight of each subtask must be confirmed, and then the fusion formula is used to fuse the result data. Taking into account the characteristics or attributes of the data itself, the degree of discreteness of the data, and the data scale, the fusion weight of each subtask is automatically calculated. The calculation formula for the fusion weight is: ; In the formula, is the data discreteness of the comprehensive situation data, is the data scale of the comprehensive situation data, is the fusion weight. Similarly, we can understand the relative importance of each data indicator, and then determine the fusion weight of each subtask. The fusion formula method is used to calculate the fusion result of the task output value. The fusion formula is: ; In the formula, result is the total score after fusion, n is the number of subtasks, and result i is the analysis result output of the i-th subtask, is the fusion weight of the i-th subtask.

[0066] In this embodiment, the real-time task requires real-time processing of data very quickly. Therefore, in order to ensure that the results of real-time processing data are quickly generated, there cannot be long calculations in the real-time task, otherwise once a data is delayed, the channel will be blocked. For the same data, the results processed by the real-time task and the offline task may be different. The result of the offline task is more accurate, and the result of the real-time task is an approximately correct result, and the data in the most recent period is usually the most useful. Therefore, we choose to use offline tasks and real-time tasks in a collaborative way, and use a computing framework that supports offline batch processing of data and online real-time processing of streaming data to perform processing tasks such as data reading.

[0067] In the whole process of comprehensive situation visual analysis, the process of realizing information visualization based on the original data is subdivided into processing levels according to the task type. The data needs to undergo complex processing and analysis, and finally implement visualization, which optimizes the task processing efficiency and improves the real-time and accuracy of data processing. After offline data processing and real-time data processing and analysis, as well as online computing service processing, the multi-source comprehensive situation data is transformed into comprehensive situation information that can be used for visualization. The situation data will pass through four layers, namely the Offline processing layer, the Nearline analysis layer, the Online computing layer, and the Visual information visualization layer, to complete the processing and analysis visualization process. The offline processing layer is responsible for offline processing of massive data; the Nearline analysis layer is mainly a real-time service, using streaming processing technology to process real-time data, such as Flink and Spark Streaming; the online computing layer is responsible for online computing, processing relatively simple operation logic, and ensuring low latency, such as the online service engine; the information visualization layer performs visual rendering of the situation data analysis and processing results, and intuitively and clearly displays the situation data mining information. After the data is processed at each layer, the production of raw data to visual information is completed.

[0068] Combination Fig.11 As shown, in this embodiment, the comprehensive situation information visual analysis system starts from the raw data, understands the data features hidden behind the situation data, obtains useful value information after processing, and realizes the complete analysis and drawing process from data to visualization step by step. It brings all-round situation and visual perception, allowing users to fully understand the comprehensive situation data, predict threats, and improve the ability to discover, identify, understand, analyze, and respond to security threats from a global perspective.

[0069] The above situation information visual analysis method, based on situation data and comprehensive display framework, establishes integrated management of various situation data resources such as manpower, materials, equipment facilities, network, data, and tasks. It develops situation awareness function by using big data technology and visual analysis technology, comprehensively analyzes different multi-source situation information and integrates and analyzes it, gives full play to the advantages of intelligent detection, big data deep mining and other technologies, forms comprehensive situation information with complete elements and rich content, organically combines manpower, materials, equipment facilities, network, data and tasks, forms a comprehensive situation information visualization view, and realizes comprehensive grasp and precise control of space environment situation. In order to carry out full management and comprehensive perception of the state of space environment domain resource allocation, establish precise control of space environment domain related activities, so as to efficiently carry out space environment domain resource allocation, so that the effectiveness of the execution of various related tasks can be fully guaranteed.

[0070] The situation information visual analysis device provided by the present invention is described below. The situation information visual analysis device described below and the situation information visual analysis method described above can be referenced to each other.

[0071] like Fig.12 As shown, in one embodiment, a situation information visual analysis device includes a data preprocessing module 1210 and an information visual analysis module 1220.

[0072] The data preprocessing module 1210 is used to obtain a data source by monitoring the data in the message consumption queue through the computing engine, and to clean and preprocess the data source to store the cleaned and preprocessed data source in a database.

[0073] The information visual analysis module 1220 is used to call the algorithm platform to perform calculations and in-depth analysis on the data sources in the database to obtain a situation information portrait, and store the situation information portrait in the data portrait information database so as to retrieve the situation information portrait in real time and map it to the information visual layer.

[0074] Among them, the data sources include multi-type comprehensive situation data from multiple sources, the comprehensive situation data includes structured data and unstructured data, and the algorithm platform is built based on multiple basic algorithms and machine learning frameworks. The basic algorithms include at least text mining algorithms, classification algorithms, regression algorithms, clustering algorithms and recommendation algorithms.

[0075] In this embodiment, the situation information visual analysis device provided by the present invention, the data preprocessing module is specifically used for: Unstructured data is extracted from the data source, and data quality filtering is performed on the unstructured data through the defined filtering model or filtering rules to filter out low-quality data in the unstructured data whose data quality does not meet the set requirements.

[0076] The Bloom filter technology is called to deduplicate the unstructured data after filtering to obtain the preprocessed unstructured data, and the structured data and the preprocessed unstructured data are stored in the database.

[0077] In this embodiment, the situation information visual analysis device provided by the present invention further includes an algorithm platform construction module for: The algorithm platform is built based on text mining algorithm, classification algorithm, regression algorithm, clustering algorithm and recommendation algorithm combined with the machine learning framework Flink.

[0078] The algorithm platform is used to abstract the calculation and deep analysis algorithms corresponding to multiple types of comprehensive situation data from multiple sources, so as to optimize the algorithm platform and obtain the calculation framework for each type of comprehensive situation data.

[0079] In this embodiment, the situation information visual analysis device provided by the present invention, the information visual analysis module is specifically used for: The algorithm platform is called to establish an analysis and processing model corresponding to each type of comprehensive situation data based on different types of comprehensive situation data.

[0080] By analyzing the processing model, multiple processing and analysis tasks corresponding to the same type of comprehensive situation data are fitted to obtain the processing and analysis results.

[0081] Among them, data is shared between the same type of comprehensive situation data, and each processing and analysis task corresponding to the same type of comprehensive situation data has shared parameters, and the processing and analysis results are obtained by fusing the analysis result outputs of multiple subtasks of each processing and analysis task.

[0082] In this embodiment, the situation information visual analysis device provided by the present invention, the information visual analysis module is specifically used for: The fusion weight of each subtask in the processing and analysis task is determined according to the data discreteness and data scale of the corresponding type of comprehensive situation data, and the fusion formula is called to fuse the analysis result outputs of multiple subtasks according to the fusion weight of each subtask.

[0083] The calculation formula of the fusion weight is: ; In the formula, is the data discreteness of the comprehensive situation data, is the data scale of the comprehensive situation data, is the fusion weight.

[0084] The fusion formula is: ; In the formula, result is the total score after fusion, n is the number of subtasks, and result i is the analysis result output of the i-th subtask, is the fusion weight of the i-th subtask.

[0085] In this embodiment, the situation information visual analysis device provided by the present invention, the information visual analysis module is specifically used for: Obtain comprehensive situation data from multiple sources and divide the comprehensive situation data into offline data and real-time data.

[0086] The offline processing layer and near-line analysis layer in the algorithm platform are called to process and analyze offline data and real-time data, and the online computing layer is called to perform calculations and in-depth analysis on the processed and analyzed offline data and real-time data to generate a situation information portrait.

[0087] Among them, the algorithm platform includes offline processing layer, near-line analysis layer and online computing layer.

[0088] In this embodiment, the situation information visual analysis device provided by the present invention, the information visual analysis module is specifically used for: The situation information portrait generated by the online computing layer is retrieved in real time from the data portrait information database and sent to the information visualization layer of the algorithm platform to map the situation information portrait to the information visualization layer.

[0089] Fig.13 An example of a physical structure diagram of an electronic device is shown. The electronic device may be a smart terminal, and its internal structure diagram may be as follows: Fig.13As shown. The electronic device includes a processor, a memory and a network interface connected through a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for visual analysis of situation information is implemented, and the method includes: The computing engine obtains the data source by monitoring the data in the message consumption queue, and cleans and pre-processes the data source to store the cleaned and pre-processed data source in the database; Call the algorithm platform to calculate and deeply analyze the data sources in the database to obtain the situation information portrait, and store the situation information portrait in the data portrait information database to retrieve the situation information portrait in real time and map it to the information visualization layer; Among them, the data sources include multi-type comprehensive situation data from multiple sources, the comprehensive situation data includes structured data and unstructured data, and the algorithm platform is built based on multiple basic algorithms and machine learning frameworks. The basic algorithms include at least text mining algorithms, classification algorithms, regression algorithms, clustering algorithms and recommendation algorithms.

[0090] Those skilled in the art will understand that Fig.13 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present invention, and does not constitute a limitation on the electronic device to which the scheme of the present invention is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0091] On the other hand, the present invention also provides a computer storage medium storing a computer program, which implements a situation information visual analysis method when the computer program is executed by a processor, the method comprising: The computing engine obtains the data source by monitoring the data in the message consumption queue, and cleans and pre-processes the data source to store the cleaned and pre-processed data source in the database; Call the algorithm platform to calculate and deeply analyze the data sources in the database to obtain the situation information portrait, and store the situation information portrait in the data portrait information database to retrieve the situation information portrait in real time and map it to the information visualization layer; Among them, the data sources include multi-type comprehensive situation data from multiple sources, the comprehensive situation data includes structured data and unstructured data, and the algorithm platform is built based on multiple basic algorithms and machine learning frameworks. The basic algorithms include at least text mining algorithms, classification algorithms, regression algorithms, clustering algorithms and recommendation algorithms.

[0092] In another aspect, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and when the processor executes the computer instructions, a method for visually analyzing situation information is implemented, the method comprising: The computing engine obtains the data source by monitoring the data in the message consumption queue, and cleans and pre-processes the data source to store the cleaned and pre-processed data source in the database; Call the algorithm platform to calculate and deeply analyze the data sources in the database to obtain the situation information portrait, and store the situation information portrait in the data portrait information database to retrieve the situation information portrait in real time and map it to the information visualization layer; Among them, the data sources include multi-type comprehensive situation data from multiple sources, the comprehensive situation data includes structured data and unstructured data, and the algorithm platform is built based on multiple basic algorithms and machine learning frameworks. The basic algorithms include at least text mining algorithms, classification algorithms, regression algorithms, clustering algorithms and recommendation algorithms.

[0093] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory.

[0094] By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0095] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0096] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A situation information visual analysis method, characterized in that: The method comprises: The computing engine obtains a data source by monitoring data in a message consumption queue, and cleans and pre-processes the data source, so as to store the cleaned and pre-processed data source in a database; Calling the algorithm platform to calculate and deeply analyze the data source in the database to obtain a situation information portrait, and storing the situation information portrait in the data portrait information database, so as to retrieve the situation information portrait in real time and map it to the information visualization layer; The data source includes multi-type comprehensive situation data from multiple sources, the comprehensive situation data includes structured data and unstructured data, the algorithm platform is constructed based on multiple basic algorithms and machine learning frameworks, and the basic algorithms include at least text mining algorithms, classification algorithms, regression algorithms, clustering algorithms, and recommendation algorithms; The calling algorithm platform calculates and deeply analyzes the data source in the database to obtain a situation information portrait, and stores the situation information portrait in a data portrait information database, so as to retrieve the situation information portrait in real time and map it to the information visualization layer, including: Calling the algorithm platform to establish an analysis and processing model corresponding to each type of the comprehensive situation data according to different types of the comprehensive situation data; Fitting a plurality of processing and analysis tasks corresponding to the same type of the comprehensive situation data through the analysis and processing model to obtain processing and analysis results; Among them, data is shared between the same type of comprehensive situation data, and each processing and analysis task corresponding to the same type of comprehensive situation data has shared parameters, and the processing and analysis results are obtained by fusing the analysis result outputs of multiple subtasks of each processing and analysis task.

2. The situation information visual analysis method according to claim 1, characterized in that: The computing engine obtains a data source by monitoring data in a message consumption queue, and cleans and pre-processes the data source to store the cleaned and pre-processed data source in a database, including: Extracting the unstructured data from the data source, and performing data quality filtering on the unstructured data through a defined filtering model or filtering rule, so as to filter out low-quality data in the unstructured data whose data quality does not meet set requirements; The Bloom filter technology is called to perform data deduplication on the unstructured data after filtering to obtain the preprocessed unstructured data, and the structured data and the preprocessed unstructured data are stored in the database.

3. The situation information visual analysis method according to claim 1, characterized in that: The calling algorithm platform calculates and deeply analyzes the data source in the database to obtain a situation information portrait, and stores the situation information portrait in a data portrait information database to retrieve the situation information portrait in real time and map it to the information visualization layer, which includes: The algorithm platform is constructed based on the text mining algorithm, classification algorithm, regression algorithm, clustering algorithm and recommendation algorithm combined with the machine learning framework Flink; The algorithm platform is used to abstract the calculation and in-depth analysis algorithms corresponding to the multiple types of comprehensive situation data from the multiple sources, so as to optimize the algorithm platform and obtain the calculation framework of each type of the comprehensive situation data.

4. The situation information visual analysis method according to claim 1, characterized in that: The calling algorithm platform calculates and deeply analyzes the data source in the database to obtain a situation information portrait, and stores the situation information portrait in a data portrait information database to retrieve the situation information portrait in real time and map it to the information visual layer, and also includes: Determine the fusion weight of each subtask in the processing and analysis task according to the data discreteness and data scale of the corresponding type of the comprehensive situation data, and call the fusion formula to fuse the analysis result outputs of multiple subtasks according to the fusion weight of each subtask; The calculation formula of the fusion weight is: ; In the formula, is the data discreteness of the comprehensive situation data, is the data scale of the comprehensive situation data, is the fusion weight; The fusion formula is: ； In the formula, result is the total score after fusion, n is the number of subtasks, and result i is the analysis result output of the i-th subtask, is the fusion weight of the i-th subtask.

5. The situation information visual analysis method according to claim 4, characterized in that: The calling algorithm platform calculates and deeply analyzes the data source in the database to obtain a situation information portrait, and stores the situation information portrait in a data portrait information database to retrieve the situation information portrait in real time and map it to the information visual layer, and also includes: Acquire the comprehensive situation data from multiple sources, and divide the comprehensive situation data into offline data and real-time data; Calling the offline processing layer and the near-line analysis layer in the algorithm platform to process and analyze the offline data and the real-time data, and calling the online computing layer to perform calculation and in-depth analysis on the processed and analyzed offline data and the real-time data to generate the situation information portrait; The algorithm platform includes the offline processing layer, the near-line analysis layer and the online calculation layer.

6. The situation information visual analysis method according to claim 5, characterized in that: The algorithm platform also includes the information visualization layer; The calling algorithm platform calculates and deeply analyzes the data source in the database to obtain a situation information portrait, and stores the situation information portrait in a data portrait information database to retrieve the situation information portrait in real time and map it to the information visual layer, and also includes: The situation information portrait generated by the online computing layer is retrieved in real time from the data portrait information database and sent to the information visualization layer of the algorithm platform to map the situation information portrait to the information visualization layer.

7. A situation information visual analysis device, characterized in that: The device comprises: A data preprocessing module is used to obtain a data source by monitoring the data in the message consumption queue through a computing engine, and to clean and preprocess the data source to store the cleaned and preprocessed data source in a database; An information visual analysis module is used to call the algorithm platform to perform calculation and in-depth analysis on the data source in the database to obtain a situation information portrait, and store the situation information portrait in a data portrait information database so as to retrieve the situation information portrait in real time and map it to the information visual layer; The data source includes multi-type comprehensive situation data from multiple sources, the comprehensive situation data includes structured data and unstructured data, the algorithm platform is constructed based on multiple basic algorithms and machine learning frameworks, and the basic algorithms include at least text mining algorithms, classification algorithms, regression algorithms, clustering algorithms, and recommendation algorithms; The information visual analysis module is specifically used for: Calling the algorithm platform to establish an analysis and processing model corresponding to each type of the comprehensive situation data according to different types of the comprehensive situation data; Fitting a plurality of processing and analysis tasks corresponding to the same type of the comprehensive situation data through the analysis and processing model to obtain processing and analysis results; Among them, data is shared between the same type of comprehensive situation data, and each processing and analysis task corresponding to the same type of comprehensive situation data has shared parameters, and the processing and analysis results are obtained by fusing the analysis result outputs of multiple subtasks of each processing and analysis task.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images